Context

The PR I made revolves around std::fs::create_dir_all() function. To quote what the function does:

Recursively create a directory and all of its parent components if they are missing.

This function is not atomic. If it returns an error, any parent components it was able to create will remain.

If the empty path is passed to this function, it always succeeds without creating any directories.

It’s essentially similar to what the mkdir -p <path> command does in the terminal.

Funnily enough, the reason why I made this PR was because of a PR I oversaw in uutils’ coreutils repository on fixing a stack overflow issue with mkdir -p on deeply nested directories. Prior to this change for the mkdir command, uutils’ coreutils manually recursed up to the first non-existing path component and used std::fs::create_dir to create each missing directory:

// Return true if the directory at `path` has been created by this call.
// `is_parent` argument is not used on windows
#[allow(unused_variables)]
fn create_dir(path: &Path, is_parent: bool, config: &Config) -> UResult<()> {
    let path_exists = path.exists();
    ...
    if path == Path::new("") {
        return Ok(());
    }

    if config.recursive {
        match path.parent() {
            Some(p) => create_dir(p, true, config)?,
            None => {
                USimpleError::new(1, translate!("mkdir-error-failed-to-create-tree"));
            }
        }
    }

    match std::fs::create_dir(path) {
        Ok(()) => {
            ...
            Ok(())
        }
        Err(_) if path.is_dir() => Ok(()),
        Err(e) => Err(e.into()),
    }
}

As naoNao89 pointed out in the PR, the implementation of uutils’ mkdir command had room for a stack overflow issue with nested directories (in his testing, the overflow occurred at 200+ nested directories of “/a” repeated). The solution to this problem is to change the implementation of mkdir -p from creating directories and missing components from recursive to iterative; from allocating the path references in the stack to the heap.

Still, when I first took a look at how uutils’ initially implemented the mkdir command with -p flag set, I thought “why didn’t they just use std::fs::create_dir_all() instead of std::fs::create_dir()?” to create all the directory components with the recursive flag set. There would be no need to re-invent the wheel on creating the given directory and its missing parent components. On second glance, however, it clicked for me. For mkdir, it has other config options like permission bits to set on the directory once it is created. If std::fs::create_dir_all() were to report an error but successfully created some directories, then it could leave the directories in an incorrect state if config options like permission bits were set.

But surely, ignoring the config options for a moment, std::fs::create_dir_all() would have avoided this stack overflow issue, right?

Rust’s std::fs::create_dir_all() Implementation

That would be an incorrect assumption to make! The following is the implementation of Rust’s create_dir_all():

path#[stable(feature = "rust1", since = "1.0.0")]
pub fn create_dir_all<P: AsRef<Path>>(path: P) -> io::Result<()> {
    DirBuilder::new().recursive(true).create(path.as_ref())
}

impl DirBuilder {
    ...
    #[stable(feature = "dir_builder", since = "1.6.0")]
    pub fn recursive(&mut self, recursive: bool) -> &mut Self {
        self.recursive = recursive;
        self
    }

    #[stable(feature = "dir_builder", since = "1.6.0")]
    pub fn create<P: AsRef<Path>>(&self, path: P) -> io::Result<()> {
        self._create(path.as_ref())
    }

    fn _create(&self, path: &Path) -> io::Result<()> {
        if self.recursive { self.create_dir_all(path) } else { self.inner.mkdir(path) }
    }

    fn create_dir_all(&self, path: &Path) -> io::Result<()> {
        if path == Path::new("") {
            return Ok(());
        }

        match self.inner.mkdir(path) {
            Ok(()) => return Ok(()),
            Err(ref e) if e.kind() == io::ErrorKind::NotFound => {}
            Err(_) if path.is_dir() => return Ok(()),
            Err(e) => return Err(e),
        }
        match path.parent() {
            Some(p) => self.create_dir_all(p)?,
            None => {
                return Err(io::const_error!(
                    io::ErrorKind::Uncategorized,
                    "failed to create whole tree",
                ));
            }
        }
        match self.inner.mkdir(path) {
            Ok(()) => Ok(()),
            Err(_) if path.is_dir() => Ok(()),
            Err(e) => Err(e),
        }
    }
    ...
}

The standard library implementation of Rust’s create_dir_all(), as promised by the doc comments, uses recursion!

What’s Wrong with Recursion Here?

For all this talk about recursion causing issues, I guess I should go a bit deeper on why recursion may not be so pleasant for this function.

Because the nature of DirBuilder’s create_dir_all() function is recursive, &self and path arguments are stored onto the stack frame repeatedly since it’s possible that the parent directories of the given path needs to be created before the current path can be created; it’s necessary to remember the paths you need to create, hence the allocation is required. This also means Tail Call Optimization (TCO) is not possible here, which means that the recursive block of code cannot be transformed into an iterative version to prevent allocating a new stack frame.

If we were to follow conventional path size limits (filesystems inherently do not have limits on how deeply nested a path is), then recursively creating directories would not pose an issue on:

• Linux-based OSs, which often has a default stack size of 8 MiB and a PATH_MAX of 4096 bytes;

• MacOS, which has a default stack size of 8 MiB and maximum path of 1024 bytes;

• Windows OS, which has a default stack size of 1 MB and a maximum path of 260 bytes

However, if we consider enabling long paths on Windows OS and multithreading across all OSs, then we would need to be aware of the following:

• Linux-based OSs tends to to default to 8 MB per thread;

• MacOS defaults to 512 KB stacks per thread (unrelating to main thread);

• And Windows OS allows paths up to 32767 bytes with long path enabled and defaults to 1 MB stacks per thread

On MacOS and Windows OS, it’s definitely possible for a potential stack overflow to occur through multithreaded code or with long paths on Windows. In fact, if we take this code

use std::fs;

fn main() {
    let mut dir_path = "".to_string();

    for letter in 0..16000 {
        dir_path = format!("{}{}", dir_path, "/a");
    }
    fs::create_dir_all(&dir_path);
}

and output the debug build assembly in Rust’s playground for std::fs::create_dir_all()

std::fs::DirBuilder::create:
    subq    $72, %rsp
    movq    %rdi, 8(%rsp)
    movq    %rsi, 32(%rsp)
    movq    %rdx, 40(%rsp)
    movq    %rdi, 48(%rsp)
    leaq    32(%rsp), %rdi
    callq    <&T as core::convert::AsRef<U>>::as_ref
    movq    %rdx, 16(%rsp)
    movq    %rax, 24(%rsp)
    jmp    .LBB1_3

.LBB1_1:
    movq    56(%rsp), %rdi
    callq    _Unwind_Resume@PLT
    movq    %rax, %rcx
    movl    %edx, %eax
    movq    %rcx, 56(%rsp)
    movl    %eax, 64(%rsp)
    jmp    .LBB1_1

.LBB1_3:
    movq    16(%rsp), %rdx
    movq    24(%rsp), %rsi
    movq    8(%rsp), %rdi
    movq    std::fs::DirBuilder::_create@GOTPCREL(%rip), %rax
    callq    *%rax
    movq    %rax, (%rsp)
    jmp    .LBB1_4

.LBB1_4:
    movq    (%rsp), %rax
    addq    $72, %rsp
    retq

std::fs::create_dir_all:
    subq    $72, %rsp
    movq    %rdi, 32(%rsp)
    movb    $1, 55(%rsp)
    movl    $511, 44(%rsp)
    movb    $0, 48(%rsp)
    movb    $1, 48(%rsp)
    leaq    32(%rsp), %rdi
    callq    <&T as core::convert::AsRef<U>>::as_ref
    movq    %rdx, 16(%rsp)
    movq    %rax, 24(%rsp)
    jmp    .LBB2_3

.LBB2_1:
    movq    56(%rsp), %rdi
    callq    _Unwind_Resume@PLT
    movq    %rax, %rcx
    movl    %edx, %eax
    movq    %rcx, 56(%rsp)
    movl    %eax, 64(%rsp)
    jmp    .LBB2_1

.LBB2_3:
    movq    16(%rsp), %rdx
    movq    24(%rsp), %rsi
    leaq    44(%rsp), %rdi
    callq    std::fs::DirBuilder::create
    movq    %rax, 8(%rsp)
    jmp    .LBB2_4

.LBB2_4:
    movq    8(%rsp), %rax
    addq    $72, %rsp
    retq

From the instruction subq $72, %rsp, we can observe the following that on Linux systems, each recursive call to std::fs::create_dir_all() takes up 72 bytes of the stack frame. Without inspecting deeply into assembly output, we can infer that this stack allocation is reserved for things like a reference to a path to mkdir on, some permission bits on creation of a directory (as 511 in decimal is 777 in octal, which is used to give rwx permissions across owner, group, and others), and return addresses. Considering shadow space padding on Windows, each recursive call to create_dir_all() could easily take up in the ballpark of ~72-112 bytes. That means on Windows, if we were to try and create a path of “/a” repeated, the stack would overflow around ~8929-13889 function calls deep. In practical cases, it’s very rare to reach this deep in this function, but it exists in the realm of possibility!

The Fix to std::fs::create_dir_all()

The key to solving this issue is just like what was proposed in naoNao89’s PR: just make the function create directories iteratively. To be precise, we should turn DirBuilder’s create_dir_all() into an iterative function. The following is the implementation change to DirBuilder’s create_dir_all():

   fn create_dir_all(&self, path: &Path) -> io::Result<()> {
        // if path's parent is None, it is "/" path, which should
        // return Ok immediately
        if path == Path::new("") || path.parent() == None {
            return Ok(());
        }

        let ancestors = path.ancestors();
        let mut uncreated_dirs = 0;

        for ancestor in ancestors {
            // for relative paths like "foo/bar", the parent of
            // "foo" will be "" which there's no need to invoke
            // a mkdir syscall on
            if ancestor == Path::new("") || ancestor.parent() == None {
                break;
            }

            match self.inner.mkdir(ancestor) {
                Ok(()) => break,
                Err(e) if e.kind() == io::ErrorKind::NotFound => uncreated_dirs += 1,
                // we check if the err is AlreadyExists for two reasons
                //    - in case the path exists as a *file*
                //    - and to avoid calls to .is_dir() in case of other errs
                //      (i.e. PermissionDenied)
                Err(e) if e.kind() == io::ErrorKind::AlreadyExists && ancestor.is_dir() => break,
                Err(e) => return Err(e),
            }
        }

        // collect only the uncreated directories w/o letting the vec resize
        let mut uncreated_dirs_vec = Vec::with_capacity(uncreated_dirs);
        uncreated_dirs_vec.extend(ancestors.take(uncreated_dirs));

        for uncreated_dir in uncreated_dirs_vec.iter().rev() {
            if let Err(e) = self.inner.mkdir(uncreated_dir) {
                if e.kind() != io::ErrorKind::AlreadyExists || !uncreated_dir.is_dir() {
                    return Err(e);
                }
            }
        }

        Ok(())
    }

In order to preserve the behavior and time complexity of recursive version of create_dir_all() to the iterative version, I had to keep in mind that:

• Root path (“/”) and empty string (““) must just return with an Ok(())

• We must iterate upward to the first non-existing path to create and then we can iterate downward to create the missing path components

• The errors returned by self.inner.mkdir() produced have different effects.

  • io::ErrorKind::NotFound tells us we need to create a missing parent component first

  • io::ErrorKind::AlreadyExists && ancestor.is_dir() tells us that we have reached the first ancestor directory that exists and we can start building the missing components there

  • Every other error is a valid error we should return impromptu

• We should only allocate onto the heap the required space to contain the remaining paths that we need to create

The last point is pretty subtle on why it’s necessary (or more so efficient than necessary). In the recursive implementation of create_dir_all(), you can see that no extra allocation or re-copying is done for the path and parent components. If we used an empty Vec and kept pushing the paths that we need to mkdir on repeatedly, we would need to resize the Vec repeatedly to allow for more paths to be pushed onto the Vec; there would be unnecessary re-allocations of heap memory occurring here. Hence, a counter variable is used to track how many directories we have left to create, and we can then take from the ancestor iterator the exact number of directories and place that onto our Vec.

In an ideal world, we can make this function better and not need to use heap allocation for the remaining directories we need to create. We could just use the ancestor iterator directly and with the counter variable we can just create the given path and missing parent directories in a forward manner. But unfortunately, the Ancestor<‘_> iterator returned by .ancestors() is not a reversible iterator; repeatedly going backward from the iterator to create the missing directories would result in a O(N²) time complexity instead of O(N). That said, this implementation matches the time and space complexity of the recursive version of create_dir_all().

Afterwords

To be honest, it’s a bit surreal that a PR that I made got merged into the Official Rust Repo. I’ve only been programming in Rust as my primary language since summer 2024 (prior to that I mostly worked with Python and C since 2021), so there’s a lot of things that I still need to learn and get comfortable with, like lifetimes and proc macro. On top of that, I started contributing to open source software/projects during last October (uutils’ coreutils being my first!), so a lot of conventions of following contributing guidelines and communicating with people through GitHub were really new to me. So when I made this PR back in October, I thought a change like this might not be looked at or be merged in.

But seeing that this PR got merged in, it made me happy. I’m happy that the code I write can have impact on people— that I could even cause that sort of impact. I’m really glad to be a part of the open source community, and especially be a part of Rust community who have all been super cool to interact with and inspiring to see what projects they’ve worked on.

I hope to be able to contribute more to the open source community and write more blog posts!

Informal Citations

• Default Stack Sizes Across Linux OSs, MacOS, and Windows OS