Home

File and I/O Abstractions

Subpage of Operating Systems

Diving deep into the design and structure of Operating Systems.

Everything is a file - The UNIX time-sharing system (1974)

There is an identical interface for dealing with:

  • Files on disk
  • Devices
  • Regular files on disk
  • Networking (sockets)
  • Local interprocess communication

All of these work with the system calls open(), read(), write() and close() (and a custom configuration ioctl()).

The abstraction of everything as a file considers the file as just a collection of data. It contains:

  1. File Data
  2. File Metadata

Files are organized in directories which are the organization structure to deal with files effectively. It has a path that uniquely identifies it.

Now, connecting back to the other core abstractions, every process has a current working directory.

High-Level File API

This considers the data as a stream which is an unformulated sequence of bytes with a position. An open stream is represented by a pointed to a file data structure. This is provided by stdio.h.

There are special streams that are defined implicitly when any program is executed:

  • stdin
  • stdout
  • stderr

These allow composition because you can chain processes together by connecting their streams appropriately.

Now, if we want to deal specifically with the file streams, we can do so using the methods we discuss below.

Opening and Closing

  • fopen(const char *filename, const char *mode) Opens a file and returns a FILE *. Modes include "r" (read), "w" (write, truncate), "a" (append), "r+" (read/write), etc.
  • fclose(FILE *stream) Closes the file and flushes buffers.

Reading and Writing

  • fgetc(FILE *stream) / fputc(int c, FILE *stream) Read or write a single character.
  • fgets(char *str, int n, FILE *stream) Reads a line into str (up to n-1 chars).
  • fputs(const char *str, FILE *stream) Writes a string.
  • fread(void *ptr, size_t size, size_t nmemb, FILE *stream) Reads binary data into ptr.
  • fwrite(const void *ptr, size_t size, size_t nmemb, FILE *stream) Writes binary data.

Positioning

  • fseek(FILE *stream, long offset, int whence) Moves the file pointer (SEEK_SET, SEEK_CUR, SEEK_END).
  • ftell(FILE *stream) Returns current position.
  • rewind(FILE *stream) Resets position to the start.

Error Handling

  • feof(FILE *stream) Tests for end-of-file.
  • ferror(FILE *stream) Tests for errors.
  • clearerr(FILE *stream) Clears EOF and error indicators.

Buffering

  • fflush(FILE *stream) Forces buffered output to be written.
  • setbuf(FILE *stream, char *buf) / setvbuf(FILE *stream, char *buf, int mode, size_t size) Control buffering behavior.

Here’s a practical example that demonstrates many of these functions:

#include <stdio.h>
#include <stdlib.h>

int main(void) {
    FILE *src, *dest;
    char buffer[256];

    // Open source file for reading
    src = fopen("input.txt", "r");
    if (src == NULL) { // Always check for NULL!
        perror("Error opening input file");
        return EXIT_FAILURE;
    }

    // Open destination file for writing
    dest = fopen("output.txt", "w");
    if (dest == NULL) { // Always check for NULL!
        perror("Error opening output file");
        fclose(src);
        return EXIT_FAILURE;
    }

    // Copy line by line
    while (fgets(buffer, sizeof(buffer), src) != NULL) {
        fputs(buffer, dest);
    }

    // Check for errors
    if (ferror(src)) {
        fprintf(stderr, "Error reading input file\n");
    }
    if (ferror(dest)) {
        fprintf(stderr, "Error writing output file\n");
    }

    // Close files
    fclose(src);
    fclose(dest);

    printf("File copy completed successfully.\n");
    return EXIT_SUCCESS;
}

Low-Level File API

When we are operating one level lower, we need to keep the following design idea in mind about the way the system is built. When it comes to UNIX, we have the following design concepts:

  • Uniformity: Everything is a file
  • A file must be opened before use
  • Addressing is always in bytes (even if we are operating at some other level such as blocks or words)
  • Kernel Buffered Reads and Writes (this will be internal to the kernel to yield performance and to match the structure of data the user is asking for)

The low-level API is where you interact directly with the operating system through system calls rather than the buffered FILE * interface.

The difference is that the low-level API does not do fancy user level actions, for example, to see if there is some buffering that can be exploited. The high-level API will thus be faster in the long term (a lot faster because system calls are 25x more expensive than function calls). This however can make it unpredictable due to the complex user level buffering logic.

For example, you need to think about when the data you write is flushed to memory from the buffer. If you need an assurance that data is written, you must use fflush with the high-level file API.

Here are the core system calls we use:

open

int open(const char *pathname, int flags, mode_t mode);
  • Opens a file and returns a file descriptor (an integer).
  • flags specify how to open: O_RDONLY, O_WRONLY, O_RDWR, O_CREAT, O_TRUNC, O_APPEND, etc.
  • mode sets permissions (used only if creating a file).
  • On success: returns a non-negative integer (file descriptor).
  • On failure: returns 1 and sets errno.

creat

int creat(const char *pathname, mode_t mode);
  • Simplified version of open that always creates/truncates a file for writing.
  • Equivalent to open(pathname, O_WRONLY | O_CREAT | O_TRUNC, mode).
  • Returns a file descriptor or 1 on error.

close

int close(int fd);
  • Closes the file descriptor, releasing OS resources.
  • Returns 0 on success, 1 on error.

These are some of the design concepts that you need to keep in mind behind this API:

  1. Integer File Descriptors
    • The OS represents open files as entries in a per-process file descriptor table.
    • Each open file gets a small integer index (0, 1, 2 are reserved for stdin, stdout, stderr).
    • Using integers makes system calls fast, lightweight, and easy to pass around. It also improves security because it prevents it means we don’t mess with the real address of the object.
  2. Minimal Abstraction
    • Unlike FILE *, there’s no buffering. You work directly with the kernel.
    • This design gives precise control over reads/writes, positioning, and concurrency.
  3. Error Handling via Return Values
    • Returning 1 on error is simple and consistent.
    • errno provides detailed error codes (e.g., EACCES, ENOENT).
    • This separation keeps the API lean while still informative.
  4. Portability Across UNIX-like Systems
    • The POSIX standard ensures open, creat, and close behave consistently across Linux, BSD, macOS, etc.
  5. Resource Management
    • Explicit close ensures you release kernel resources.
    • Prevents leaks and exhaustion of the per-process file descriptor limit.
  6. Kernel Buffering
    • Reads and writes are all buffered inside the kernel and this is part of global buffer management and caching.

Here is a simple example of creating and writing a file:

#include <fcntl.h>   // for open, creat
#include <unistd.h>  // for write, close
#include <stdio.h>   // for perror
#include <string.h>  // for strlen

int main(void) {
    int fd;
    const char *msg = "Hello, low-level I/O!\n";

    // Create or open file with write-only access
    fd = open("lowlevel.txt", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    if (fd == -1) {
        perror("open");
        return 1;
    }

    // Write to file
    if (write(fd, msg, strlen(msg)) == -1) {
        perror("write");
        close(fd);
        return 1;
    }

    // Close file
    if (close(fd) == -1) {
        perror("close");
        return 1;
    }

    return 0;
}

The low-level API (open, creat, close, plus read/write) is designed for speed, simplicity, and direct OS control.

Low-Level I/O API

Here, we have the following additional classe of operations we work:

Control & Metadata

  • fcntl(fd, cmd, …) → Manipulate descriptor flags (locking, non-blocking).
  • ioctl(fd, request, …) → Device-specific control operations (e.g., terminal settings, disk operations).
  • stat(path, struct stat *buf) → Retrieve file metadata.

ioctl is a catch-all: it extends the I/O model beyond simple byte streams, allowing direct communication with devices. Thus, it’s more specialized — often used for device drivers and system-level configuration rather than everyday file manipulation.

Process State

On a successful call to open() we will have a file descriptor returned to the user. At the same time, an open file description is created in the kernel. For each process, the kernel maintains a mapping from the file descriptor to the open file file description called the file descriptor table.

So here are the two levels of abstraction we are working with:

  1. File Descriptor (FD)
    • A small integer returned to the process by open().
    • It’s just an index into the process’s file descriptor table.
    • Each process has its own table, so descriptors are local to that process.
  2. Open File Description (OFD)
    • A kernel-level structure created when a file is opened.
    • Contains the actual state of the open file:
      • Current file offset (cursor position).
      • Access mode (read, write, append).
      • Status flags (non-blocking, synchronous writes, etc.).
      • Reference to the underlying inode (the file’s metadata).
    • Multiple file descriptors (even across processes) can point to the same OFD if they are duplicated (we can duplicate using dup, we create a new process using fork, etc.).
    • They don’t belong to processes. Thus, they can be easily shared across processes.

When you call open(), the kernel creates a new open file description. It then adds an entry in your process’s file descriptor table, mapping the returned integer FD → that OFD.

When you call dup() or fork() the new FD points to the same OFD, meaning they share file offset and flags. When you call close(fd), the FD entry is removed from the table. If no other FD points to the OFD, the kernel destroys the OFD and releases resources.

This gives the following advantages:

  • Efficiency: Integers are lightweight handles, easy to store and pass.
  • Isolation: Each process has its own FD table, but OFDs can be shared when needed.
  • Flexibility: Sharing OFDs enables inter-process communication and coordinated file access.
  • Consistency: The separation ensures that the kernel tracks file state centrally, while processes only manipulate descriptors.

Note that when a process is forked, the file descriptor table is copied over so we can still access the open file descriptions from the parent process in the child process.

The OFD that the table point to is the same. Aliasing OFDs is a good idea because it allows us to share resources effectively. This is great in POSIX because all I/O is abstracted as a file operation.

One implication is that all processes has the same OFDs as the shell so that all the outputs from the processes started by the shell go back to the terminal. And because we are aliasing, if one process closes its connection, the others are not affected.

/ Continue

Follow the technical trail.

Use the dense notes as the source material, then move through the guided route, writing, or project proof when you want a cleaner entry point.

Start Here Writing Projects Subscribe