Merge pull request #1447 from trapexit/branchcfg
Some checks failed
CodeQL / Analyze (cpp) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
mkdocs / deploy (push) Has been cancelled

Improve mount waiting + misc doc improvements
This commit is contained in:
trapexit 2025-04-23 16:40:26 -05:00 committed by GitHub
commit 8ee998e8aa
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
26 changed files with 583 additions and 212 deletions

View File

@ -1,18 +1,20 @@
# branches # branches
The 'branches' argument is a colon (':') delimited list of paths to be The `branches` argument is a colon (':') delimited list of paths to be
pooled together. It does not matter if the paths are on the same or pooled together. It does not matter if the paths are on the same or
different filesystems nor does it matter the filesystem type (within different filesystems nor does it matter [the filesystem
reason). Used and available space will not be duplicated for paths on type](../faq/compatibility_and_integration.md/#what-filesystems-can-be-used-as-branches)
the same filesystem and any features which aren't supported by the (within reason). Used and available space metrics will not be
underlying filesystem (such as file attributes or extended attributes) duplicated for paths on the same filesystem and any features which
will return the appropriate errors. aren't supported by the underlying filesystem (such as file attributes
or extended attributes) will return the appropriate errors.
Branches currently have two options which can be set. A type which Branches currently have two options which can be set. A
impacts whether or not the branch is included in a policy calculation [mode](#branch-mode) which impacts whether or not the branch is
and a individual minfreespace value. The values are set by prepending included in a policy calculation and a individual
an `=` at the end of a branch designation and using commas as [minfreespace](#minfreespace) value. The values are set by
delimiters. Example: `/mnt/drive=RW,1234` prepending an `=` at the end of a branch designation and using commas
as delimiters. Example: `/mnt/drive=RW,1234`
### branch mode ### branch mode
@ -29,7 +31,7 @@ delimiters. Example: `/mnt/drive=RW,1234`
### minfreespace ### minfreespace
Same purpose and syntax as the [global option](minfreespace.md) but Same purpose and syntax as the [global option](minfreespace.md) but
specific to the branch. If not set the global value is used. specific to the branch. Defaults to the global value.
### globbing ### globbing
@ -43,8 +45,8 @@ apply the glob itself.**
# mergerfs /mnt/hdd\*:/mnt/ssd /media # mergerfs /mnt/hdd\*:/mnt/ssd /media
``` ```
The above line will use all mount points in /mnt prefixed with **hdd** The above line will use all directories in /mnt prefixed with **hdd**
and **ssd**. as well as **ssd**.
To have the pool mounted at boot or otherwise accessible from related To have the pool mounted at boot or otherwise accessible from related
tools use `/etc/fstab`. tools use `/etc/fstab`.
@ -54,4 +56,154 @@ tools use `/etc/fstab`.
/mnt/hdd*:/mnt/ssd /media mergerfs minfreespace=16G 0 0 /mnt/hdd*:/mnt/ssd /media mergerfs minfreespace=16G 0 0
``` ```
**NOTE:** The globbing is done at mount or when updated using the runtime API. If a new directory is added matching the glob after the fact it will not be automatically included. **NOTE:** The globbing is done at mount or when updated using the
runtime API. If a new directory is added matching the glob after the
fact it will not be automatically included.
A convenient way to configure branches is to use
[symlinks](../quickstart.md/#etcfstab-w-config-file).
## branch setup
mergerfs does not require any special setup of branch paths in order
to be used however here are some suggestions.
### layout
When you have a large collection of storage devices, types of storage,
and types of interconnects it can be useful to have a verbose naming
convention. My preference is to create a directory within `/mnt/` for
each major type of storage device and/or interconnect. `hdd` for
traditional hard drives; `ssd` for slower SATA based SSDs; `nvme` for
M.2 or U.2 NVME SSDs; `remote` for NFS, SMB, sshfs, etc.
The mount points within those directories are named after the size of
the storage and its serial number in the case of physical storage and
the hostname and remote filesystem for remote.
```
$ ls -lh /mnt/
total 16K
drwxr-xr-x 8 root root 4.0K Aug 18 2024 hdd
drwxr-xr-x 6 root root 4.0K Oct 8 2024 nvme
drwxr-xr-x 3 root root 4.0K Aug 24 2024 remote
drwxr-xr-x 3 root root 4.0K Jul 14 2024 ssd
$ ls -lh /mnt/hdd/
total 8K
d--------- 2 root root 4.0K Apr 14 15:58 10T-01234567
d--------- 2 root root 4.0K Apr 12 20:51 20T-12345678
$ ls -lh /mnt/nvme/
total 8K
d--------- 2 root root 4.0K Apr 14 16:00 1T-ABCDEFGH
d--------- 2 root root 4.0K Apr 14 23:24 1T-BCDEFGHI
$ ls -lh /mnt/remote/
total 8K
d--------- 2 root root 4.0K Apr 12 20:23 foo-sshfs
d--------- 2 root root 4.0K Apr 12 20:24 bar-nfs
# You can find the serial number of a drive using lsblk
$ lsblk -d -o NAME,PATH,SIZE,SERIAL
NAME PATH SIZE SERIAL
sda /dev/sda 9.1T 01234567
sdb /dev/sdb 18.2T 12345678
nvme0n1 /dev/nvme0n1 953.9G ABCDEFGH
nvme1n1 /dev/nvme1n1 953.9G BCDEFGHI
```
### permissions and mode
#### mount points
To ensure the directory is **only** used as a point to mount another
filesystem it is good to lock it down as much as possible. Be sure to
do this **before** mounting a filesystem to it.
```
$ sudo chown root:root /mnt/hdd/10T-XYZ
$ sudo chmod 0000 /mnt/hdd/10T-XYZ
$ sudo setfattr -n user.mergerfs.branch_mounts_here
$ sudo chattr +i /mnt/hdd/10T-XYZ
```
The extended attribute `user.mergerfs.branch_mounts_here` is used by
the [branches-mount-timeout](branches_mount_timeout.md) option to
recognize whether or not a mergerfs branch path points to the intended
filesystem.
The `chattr` is likely to only work on EXT{2,3,4} filesystems but will
restrict even `root` from modifying the directory or its content.
#### mounted filesystems
For those new to Linux, intending to be the primary individual logged
into the system, or simply want to simplify permissions it is
recommended to set the root of mounted filesystems like `/tmp/` is set
to. Owned by `root`, `ugo+rwx` and [sticky
bit](https://en.wikipedia.org/wiki/Sticky_bit) set.
This must be done **after** mounting the filesystem to the target
mount point.
```
$ sudo chown root:root /mnt/hdd/10T-SERIALNUM
$ sudo chmod 1777 /mnt/hdd/10T-SERIALNUM
$ sudo setfattr -n user.mergerfs.branch /mnt/hdd/10T-SERIALNUM
```
### formatting
While even less relevant to mergerfs than the details above the topic
does com up regularly with mergerfs users. When it comes to
partitioning and formatting it is suggested to keep things
simple. Most users of mergerfs will be using the whole drive capacity
and as such have a singular partition and filesystem on that
partition. Something many people don't realize is that the partition
is not necessary and actually can become problematic.
Rather than creating paritions just format the block device. Let's use
`/dev/sda` as an example.
```
$ lsblk -d -o NAME,PATH,SERIAL /dev/sda
NAME PATH SERIAL
sda /dev/sda 01234567
$ sudo mkfs.ext4 -m0 -L 01234567 /dev/sda
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: ede8bf96-9aff-464e-a4fb-a4ab8a197cd7
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
```
You can then use the serial number as the identifier in `fstab`.
```
# <file system> <mount point> <type> <options> <dump> <pass>
LABEL=01234567 /mnt/hdd/10TB-01234567 auto nofail 0 2
```
Benefits of doing it this way?
* One less thing (partitions) to configure and worry about.
* Guarantees alignment of blocks
* Makes it easier to use the drive with different enclosures. Some
SATA to USB adapters use 4K blocks while the drive itself is using
512b. If you create partitions with one block size and move the
drive to a controller that uses the other the offset on the device
where the filesystem starts will be misinterpreted. It is possible
to manually fix this but it isn't well documented and avoidable.

View File

@ -0,0 +1,48 @@
# branches-mount-timeout
Default: `0`
mergerfs is compatible with any path, whether it be a mount point or a
directory on your [root
filesystem.](https://en.wikipedia.org/wiki/Root_directory) It doesn’t
require branch paths to be mounted or to match a specific filesystem
at startup, and it operates effectively without needing details about
the intended filesystem. This flexibility eliminates the need to
manage mount ordering, which is particularly advantageous on modern
systems where filesystems are mounted asynchronously, resulting in
unpredictable mount sequences. While
[systemd](https://www.freedesktop.org/software/systemd/man/latest/systemd.mount.html)
offers a way to enforce mount dependencies using the
[x-systemd.requires=PATH](https://www.freedesktop.org/software/systemd/man/latest/systemd.mount.html#x-systemd.requires=)
option in /etc/fstab, applying this to every branch path is both
cumbersome and susceptible to errors.
`branches-mount-timeout` will cause mergerfs to wait for all branches
to become "mounted." The logic to determine "mounted" is as follows.
1. It is mounted if the branch directory has the extended attribute
`user.mergerfs.branch`
2. It is mounted if the branch directory contains a file named
`.mergerfs.branch`
3. It is **not** mounted if the branch directory has the extended
attribute `user.mergerfs.branch_mounts_here`
4. It is **not** mounted if the branch directory contains
a file named `.mergerfs.branch_mounts_here`
5. It is mounted if the `st_dev` value of the mergerfs mountpoint is
different from the branch path.
**NOTE:** If on a `systemd` based system and using `fstab` it is a
good idea to set the mount option
[x-systemd.mount-timeout](https://www.freedesktop.org/software/systemd/man/latest/systemd.mount.html#x-systemd.mount-timeout=)
to some value longer than `branches-mount-timeout`.
## branches-mount-timeout-fail
Default: `false`
When set to `true` mergerfs will fail entirely after
`branches-mount-timeout` expires without all branches being
mounted. If set to `false` it will simply ignore the mount status of
the branches and continue on. The details will be
[logged](../error_handling_and_logging.md).

View File

@ -1,28 +1,16 @@
# functions, categories and policies # functions, categories and policies
The POSIX filesystem API is made up of a number of The POSIX filesystem API is made up of a number of functions. `creat`,
functions. **creat**, **stat**, **chown**, etc. For ease of `stat`, `chown`, etc. For ease of configuration in mergerfs, most of
configuration in mergerfs, most of the core functions are grouped into the core functions are grouped into 3 categories: `action`, `create`,
3 categories: **action**, **create**, and **search**. These functions and `search`. These functions and categories can be assigned a policy
and categories can be assigned a policy which dictates which branch is which dictates which branch is chosen when performing that
chosen when performing that function. function.
Some functions, listed in the category `N/A` below, can not be Some functions, listed in the category `N/A` below, can not be
assigned the normal policies because they are directly related to a assigned the normal policies because they are directly related to a
file which has already been opened. file which has already been opened.
When using policies which are based on a branch's available space the
base path provided is used. Not the full path to the file in
question. Meaning that mounts in the branch won't be considered in the
space calculations. The reason is that it doesn't really work for
non-path preserving policies and can lead to non-obvious behaviors.
NOTE: While any policy can be assigned to a function or category,
some may not be very useful in practice. For instance: **rand**
(random) may be useful for file creation (create) but could lead to
very odd behavior if used for `chmod` if there were more than one copy
of the file.
## Functions and their Category classifications ## Functions and their Category classifications
@ -34,43 +22,61 @@ of the file.
| N/A | fchmod, fchown, futimens, ftruncate, fallocate, fgetattr, fsync, ioctl (files), read, readdir, release, statfs, write, copy_file_range | | N/A | fchmod, fchown, futimens, ftruncate, fallocate, fgetattr, fsync, ioctl (files), read, readdir, release, statfs, write, copy_file_range |
In cases where something may be searched for (such as a path to clone) In cases where something may be searched for (such as a path to clone)
**getattr** will usually be used. `getattr` will usually be used.
## Policies ## Policies
See below for [available policies and their descriptions](#policy-descriptions).
A policy is an algorithm designed to select one or more branches for a A policy is an algorithm designed to select one or more branches for a
function to operate on. function to operate on.
Any function in the `create` category will clone the relative path if Policies do not actually manage the filesystem or layout. They are
needed. Some other functions (`rename`,`link`,`ioctl`) have special strictly responsible for deciding which files or branches will be
requirements or behaviors which you can read more about below. worked on in relation to the function being performed. Once the branch
is chosen other parts of the system does what is necessary to
accomplish the function. Such as the cloning of directories between
branches.
When using policies which are based on a branch's available space the
branch base path provided is used. Not the full path to the file or
directory in question. Meaning that mounts within the branch will not
be considered in the space calculations.
NOTE: While any policy can be assigned to a function or category, some
may not be very useful in practice. For instance: `rand` (random) may
be useful for file creation but could lead to very odd behavior if
used for `chmod` if there were more than one copy of the file. Unless
users find this flexibility useful it will likely be removed in the
future.
## Filtering ## Filtering
Most policies basically search branches and create a list of files / paths Most policies search branches and create a list of files / paths for
for functions to work on. The policy is responsible for filtering and functions to work on. The policy is responsible for filtering and
sorting the branches. Filters include **minfreespace**, whether or not sorting the branches. Filters include [minfreespace](minfreespace.md),
a branch is mounted read-only, and the branch tagging whether or not a branch is mounted read-only, and the branch mode
(RO,NC,RW). These filters are applied across most policies. (RO,NC,RW). These filters are applied across most policies.
- No **search** function policies filter. - No `search` function policies filter.
- All **action** function policies filter out branches which are - All `action` function policies filter out branches which are
mounted **read-only** or tagged as **RO (read-only)**. mounted **read-only** or mode is **RO (read-only)**.
- All **create** function policies filter out branches which are - All `create` function policies filter out branches which are
mounted **read-only**, tagged **RO (read-only)** or **NC (no mounted **read-only**, mode **RO (read-only)** or **NC (no
create)**, or has available space less than `minfreespace`. create)**, or has available space less than
[minfreespace](minfreespace.md).
Policies may have their own additional filtering such as those that Policies may have their own additional filtering such as those that
require existing paths to be present. require existing paths to be present.
If all branches are filtered an error will be returned. Typically If all branches are filtered an error will be returned. Typically
**EROFS** (read-only filesystem) or **ENOSPC** (no space left on `EROFS` (read-only filesystem) or `ENOSPC` (no space left on
device) depending on the most recent reason for filtering a device) depending on the most recent reason for filtering a
branch. **ENOENT** will be returned if no eligible branch is found. branch.
If **create**, **mkdir**, **mknod**, or **symlink** fail with `EROFS` If `create`, `mkdir`, `mknod`, or `symlink` fail with `EROFS`
or other fundamental errors then mergerfs will mark any branch found or other fundamental errors then mergerfs will mark any branch found
to be read-only as such (IE will set the mode `RO`) and will rerun the to be read-only as such (IE will set the mode `RO`) and will rerun the
policy and try again. This is mostly for `ext4` filesystems that can policy and try again. This is mostly for `ext4` filesystems that can
@ -82,16 +88,12 @@ suddenly become read-only when it encounters an error.
Policies, as described below, are of two basic classifications. `path Policies, as described below, are of two basic classifications. `path
preserving` and `non-path preserving`. preserving` and `non-path preserving`.
All policies which start with `ep` (**epff**, **eplfs**, **eplus**, All policies which start with `ep` (`epff`, `eplfs`, `eplus`, `epmfs`,
**epmfs**, **eprand**) are `path preserving`. `ep` stands for `eprand`) are `path preserving`. `ep` stands for `existing path`.
`existing path`.
A path preserving policy will only consider branches where the relative A path preserving policy will only consider branches where the relative
path being accessed already exists. path being accessed already exists.
When using non-path preserving policies paths will be cloned to target
branches as necessary.
With the `msp` or `most shared path` policies they are defined as With the `msp` or `most shared path` policies they are defined as
`path preserving` for the purpose of controlling `link` and `rename`'s `path preserving` for the purpose of controlling `link` and `rename`'s
behaviors since `ignorepponrename` is available to disable that behaviors since `ignorepponrename` is available to disable that
@ -141,5 +143,5 @@ policies is not appropriate.
| Category | Policy | | Category | Policy |
| -------- | ------ | | -------- | ------ |
| action | epall | | action | epall |
| create | epmfs | | create | pfrd |
| search | ff | | search | ff |

View File

@ -17,31 +17,30 @@ anywhere.
## Why are all my files ending up on 1 filesystem?! ## Why are all my files ending up on 1 filesystem?!
Did you start with empty filesystems? Are you using an `existing path` Did you start with empty filesystems? Are you using an `existing path`
policy? policy such as `epmfs`?
The default create policy is `epmfs`. That is a path preserving If starting with a set of empty filesystems such a policy will select
algorithm. With such a policy for `mkdir` and `create` with a set of only 1 filesystem when the first directory is created. Anything
empty filesystems it will select only 1 filesystem when the first created in that directory will be placed on the same branch because
directory is created. Anything, files or directories, created in that that is the point of such a policy. That is the expected behavior.
directory will be placed on the same branch because it is preserving
paths. That is the expected behavior.
This may catch new users off guard but this policy is the safest This may catch users off guard but the point of those `existing path`
policy to have as default as it will not change the general layout of policies is to preserve the directory layout and you need to consider
the underlying branches. If you do not care about path preservation what that means one event at a time.
([most should
not](configuration_and_policies.md#how-can-i-ensure-files-are-collocated-on-the-same-branch)) To "fix" this situation change the policy. Most users [have little
and wish your files to be spread across all your filesystems change reason](configuration_and_policies.md#how-can-i-ensure-files-are-collocated-on-the-same-branch)
`create.category` to `pfrd`, `rand`, `mfs` or a similarly non-path to use such a policy. See the [quick start guide](../quickstart.md)
restrictive [policy](../config/functions_categories_policies.md). and [FAQ](configuration_and_policies.md) for policy recommendations.
## Why do I get an "out of space" / "no space left on device" / ENOSPC error even though there appears to be lots of space available? ## Why do I get an "out of space" / "no space left on device" / ENOSPC error even though there appears to be lots of space available?
First make sure you've read the sections above about [policies, path First make sure you've read the sections about [policies, path
preservation, branch preservation, branch
filtering,](../config/functions_categories_policies.md) and the filtering,](../config/functions_categories_policies.md) and the
options `minfreespace`, [moveonenospc](../config/moveonenospc.md), options [minfreespace](../config/minfreespace.md),
[moveonenospc](../config/moveonenospc.md),
[statfs](../config/statfs.md), and [statfs](../config/statfs.md), and
[statfs_ignore](../config/statfs.md#statfs_ignore). [statfs_ignore](../config/statfs.md#statfs_ignore).

View File

@ -2,79 +2,79 @@
## Tutorials / Articles ## Tutorials / Articles
- 2016-02-02 - [Linuxserver.io: The Perfect Media Server 2016](https://blog.linuxserver.io/2016/02/02/the-perfect-media-server-2016/) * 2016-02-02 - [Linuxserver.io: The Perfect Media Server 2016](https://blog.linuxserver.io/2016/02/02/the-perfect-media-server-2016/)
- 2016-08-31 - [ZackReed.me: Mergerfs – another good option to pool your SnapRAID disks](https://zackreed.me/mergerfs-another-good-option-to-pool-your-snapraid-disks/) * 2016-08-31 - [ZackReed.me: Mergerfs – another good option to pool your SnapRAID disks](https://zackreed.me/mergerfs-another-good-option-to-pool-your-snapraid-disks/)
- 2016-11-06 - [Linuxserver.io: Revisiting the HP ProLiant Gen8 G1610T Microserver](https://blog.linuxserver.io/2016/11/06/revisiting-the-hp-proliant-gen8-g1610t-microserver/) * 2016-11-06 - [Linuxserver.io: Revisiting the HP ProLiant Gen8 G1610T Microserver](https://blog.linuxserver.io/2016/11/06/revisiting-the-hp-proliant-gen8-g1610t-microserver/)
- 2017-01-17 - [Setting up mergerfs on JBOD's (or a poor mans storage array)](http://corywestropp.com/develop/articles/setting-up-mergerfs/) * 2017-01-17 - [Setting up mergerfs on JBOD's (or a poor mans storage array)](http://corywestropp.com/develop/articles/setting-up-mergerfs/)
- 2017-06-24 - [Linuxserver.io: The Perfect Media Server 2017](https://blog.linuxserver.io/2017/06/24/the-perfect-media-server-2017/) * 2017-06-24 - [Linuxserver.io: The Perfect Media Server 2017](https://blog.linuxserver.io/2017/06/24/the-perfect-media-server-2017/)
- 2018-02-19 - [Teknophiles: Disk Pooling in Linux with mergerFS](https://web.archive.org/web/20210324184857/https://www.teknophiles.com/2018/02/19/disk-pooling-in-linux-with-mergerfs/) * 2018-02-19 - [Teknophiles: Disk Pooling in Linux with mergerFS](https://web.archive.org/web/20210324184857/https://www.teknophiles.com/2018/02/19/disk-pooling-in-linux-with-mergerfs/)
- 2018-02-20 - [Fortes.com: Using Rclone and MergerFS together across drives](https://fortes.com/2018/rclone-and-mergerfs/) * 2018-02-20 - [Fortes.com: Using Rclone and MergerFS together across drives](https://fortes.com/2018/rclone-and-mergerfs/)
- 2019-02-10 - [Medium: Migrating from ZFS to MergerFS and SnapRAID at home](https://medium.com/@pascal.brokmeier/migrating-from-zfs-to-mergerfs-and-snapraid-at-home-89c45fd5db02) * 2019-02-10 - [Medium: Migrating from ZFS to MergerFS and SnapRAID at home](https://medium.com/@pascal.brokmeier/migrating-from-zfs-to-mergerfs-and-snapraid-at-home-89c45fd5db02)
- 2019-04-24 - [MichaelXander.com: DIY NAS with OMV, SnapRAID, MergerFS, and Disk Encryption](https://michaelxander.com/diy-nas/) * 2019-04-24 - [MichaelXander.com: DIY NAS with OMV, SnapRAID, MergerFS, and Disk Encryption](https://michaelxander.com/diy-nas/)
- 2019-07-16 - [Linuxserver.io: The Perfect Media Server - 2019 Edition](https://blog.linuxserver.io/2019/07/16/perfect-media-server-2019/) * 2019-07-16 - [Linuxserver.io: The Perfect Media Server - 2019 Edition](https://blog.linuxserver.io/2019/07/16/perfect-media-server-2019/)
- 2019-09-10 - [Rclone VFS and MergerFS Setup](https://docs.usbx.me/books/rclone/page/rclone-vfs-and-mergerfs-setup) * 2019-09-10 - [Rclone VFS and MergerFS Setup](https://docs.usbx.me/books/rclone/page/rclone-vfs-and-mergerfs-setup)
- 2019-12-20 - [NetworkShinobi.com: SnapRAID and MergerFS on OpenMediaVault](https://www.networkshinobi.com/snapraid-and-mergerfs-on-openmediavault/) * 2019-12-20 - [NetworkShinobi.com: SnapRAID and MergerFS on OpenMediaVault](https://www.networkshinobi.com/snapraid-and-mergerfs-on-openmediavault/)
- 2020-01-14 - [Brandon Rozek's Blog](https://brandonrozek.com/blog/mergerfs/) * 2020-01-14 - [Brandon Rozek's Blog](https://brandonrozek.com/blog/mergerfs/)
- 2020-02-14 - [SelfHostedHome.com: Combining Different Sized Drives with mergerfs and SnapRAID](https://selfhostedhome.com/combining-different-sized-drives-with-mergerfs-and-snapraid/) * 2020-02-14 - [SelfHostedHome.com: Combining Different Sized Drives with mergerfs and SnapRAID](https://selfhostedhome.com/combining-different-sized-drives-with-mergerfs-and-snapraid/)
- 2020-05-01 - [FedoraMagazine.org: Using mergerfs to increase your virtual storage](https://fedoramagazine.org/using-mergerfs-to-increase-your-virtual-storage/) * 2020-05-01 - [FedoraMagazine.org: Using mergerfs to increase your virtual storage](https://fedoramagazine.org/using-mergerfs-to-increase-your-virtual-storage/)
- 2020-08-20 - [Setting up Rclone, Mergerfs and Crontab for automated cloud storage](https://bytesized-hosting.com/pages/setting-up-rclone-mergerfs-and-crontab-for-automated-cloud-storage) * 2020-08-20 - [Setting up Rclone, Mergerfs and Crontab for automated cloud storage](https://bytesized-hosting.com/pages/setting-up-rclone-mergerfs-and-crontab-for-automated-cloud-storage)
- 2020-11-22 - [Introducing… MergerFS – My FREE UNRAID alternative](https://supertechfreaks.com/introducing-mergerfs-free-unraid-alternative/) * 2020-11-22 - [Introducing… MergerFS – My FREE UNRAID alternative](https://supertechfreaks.com/introducing-mergerfs-free-unraid-alternative/)
- 2020-12-30 - [Perfect Media Server](https://perfectmediaserver.com) (a new site with docs fully fleshing out the 'Perfect Media Server' blog series) * 2020-12-30 - [Perfect Media Server](https://perfectmediaserver.com) (a new site with docs fully fleshing out the 'Perfect Media Server' blog series)
- 2021-07-24 - [Building the Ultimate Linux Home Server - Part 1: Intro, MergerFS, and SnapRAID](https://blog.karaolidis.com/ultimate-home-server-part-1/) * 2021-07-24 - [Building the Ultimate Linux Home Server - Part 1: Intro, MergerFS, and SnapRAID](https://blog.karaolidis.com/ultimate-home-server-part-1/)
- 2021-10-31 - [Better Home Storage: MergerFS + SnapRAID on OpenMediaVault](https://blog.sakuragawa.moe/better-home-storage-mergerfs-snapraid-on-openmediavault/) * 2021-10-31 - [Better Home Storage: MergerFS + SnapRAID on OpenMediaVault](https://blog.sakuragawa.moe/better-home-storage-mergerfs-snapraid-on-openmediavault/)
- 2021-11-28 - [Linux Magazine: Come Together - Merging file systems for a simple NAS with MergerFS](https://www.linux-magazine.com/Issues/2022/254/MergerFS) * 2021-11-28 - [Linux Magazine: Come Together - Merging file systems for a simple NAS with MergerFS](https://www.linux-magazine.com/Issues/2022/254/MergerFS)
- 2022-06-04 - [MergerFS + SnapRaid Study](https://crashlaker.github.io/2022/06/04/mergerfs_+_snapraid_study.html) * 2022-06-04 - [MergerFS + SnapRaid Study](https://crashlaker.github.io/2022/06/04/mergerfs_+_snapraid_study.html)
- 2022-12-31 - [Merge Storages in CasaOS: A secret beta feature you know now](https://blog.casaos.io/blog/13.html) * 2022-12-31 - [Merge Storages in CasaOS: A secret beta feature you know now](https://blog.casaos.io/blog/13.html)
- 2023-02-03 - [(MergerFS + SnapRAID) is the new RAID 5](https://thenomadcode.tech/mergerfs-snapraid-is-the-new-raid-5) * 2023-02-03 - [(MergerFS + SnapRAID) is the new RAID 5](https://thenomadcode.tech/mergerfs-snapraid-is-the-new-raid-5)
- 2024-02-07 - [Designing & Deploying MANS - A Hybrid NAS Approach with SnapRAID, MergerFS, and OpenZFS](https://blog.muffn.io/posts/part-3-mini-100tb-nas) * 2024-02-07 - [Designing & Deploying MANS - A Hybrid NAS Approach with SnapRAID, MergerFS, and OpenZFS](https://blog.muffn.io/posts/part-3-mini-100tb-nas)
- 2024-03-11 - [Using MergerFS to combine multiple hard drives into one unified media storage](https://fullmetalbrackets.com/blog/two-drives-mergerfs/) * 2024-03-11 - [Using MergerFS to combine multiple hard drives into one unified media storage](https://fullmetalbrackets.com/blog/two-drives-mergerfs/)
- 2024-12-20 - [Pooling multiple drives on my Raspberry Pi with mergerfs](https://sebi.io/posts/2024-12-20-pooling-multiple-drives-with-mergerfs/) * 2024-12-20 - [Pooling multiple drives on my Raspberry Pi with mergerfs](https://sebi.io/posts/2024-12-20-pooling-multiple-drives-with-mergerfs/)
## Videos ## Videos
- 2017-06-23 - [Alex Kretzschmar: Part 1 - Perfect Media Server 2017 - Introduction](https://www.youtube.com/watch?v=L5MH8q3lmmk) * 2017-06-23 - [Alex Kretzschmar: Part 1 - Perfect Media Server 2017 - Introduction](https://www.youtube.com/watch?v=L5MH8q3lmmk)
- 2017-06-24 - [Alex Kretzschmar: Part 2 - Perfect Media server 2017 - Installing Debian 9 Stretch](https://www.youtube.com/watch?v=YpVVYRN_L_A) * 2017-06-24 - [Alex Kretzschmar: Part 2 - Perfect Media server 2017 - Installing Debian 9 Stretch](https://www.youtube.com/watch?v=YpVVYRN_L_A)
- 2017-06-24 - [Alex Kretzschmar: Part 3 - Perfect Media Server 2017 - Install MergerFS and setting up your drives](https://www.youtube.com/watch?v=tbCMfm-jJ5Y) * 2017-06-24 - [Alex Kretzschmar: Part 3 - Perfect Media Server 2017 - Install MergerFS and setting up your drives](https://www.youtube.com/watch?v=tbCMfm-jJ5Y)
- 2017-06-24 - [Alex Kretzschmar: Part 4 - Perfect Media Server 2017 - Installing Docker](https://www.youtube.com/watch?v=WYI32kx4hPE) * 2017-06-24 - [Alex Kretzschmar: Part 4 - Perfect Media Server 2017 - Installing Docker](https://www.youtube.com/watch?v=WYI32kx4hPE)
- 2017-06-24 - [Alex Kretzschmar: Part 5 - Perfect Media Server 2017 - Installing and Automating SnapRAID](https://www.youtube.com/watch?v=Ir5ZsUIbHXA) * 2017-06-24 - [Alex Kretzschmar: Part 5 - Perfect Media Server 2017 - Installing and Automating SnapRAID](https://www.youtube.com/watch?v=Ir5ZsUIbHXA)
- 2017-06-24 - [Alex Kretzschmar: Part 6 - Perfect Media Server 2017 -Turning your server into a NAS with Samba and NFS](https://www.youtube.com/watch?v=1hVdWq758ZQ) * 2017-06-24 - [Alex Kretzschmar: Part 6 - Perfect Media Server 2017 -Turning your server into a NAS with Samba and NFS](https://www.youtube.com/watch?v=1hVdWq758ZQ)
- 2017-06-24 - [Alex Kretzschmar: Part 7 - Perfect Media Server 2017 - Managing your apps with docker-compose](https://www.youtube.com/watch?v=aI2rdw7_AmE) * 2017-06-24 - [Alex Kretzschmar: Part 7 - Perfect Media Server 2017 - Managing your apps with docker-compose](https://www.youtube.com/watch?v=aI2rdw7_AmE)
- 2017-06-24 - [Alex Kretzschmar: Part 8 - Perfect Media Server 2017 - Manging your server using a web UI (Cockpit and Portainer)](https://www.youtube.com/watch?v=aLyTWdzDiCg) * 2017-06-24 - [Alex Kretzschmar: Part 8 - Perfect Media Server 2017 - Manging your server using a web UI (Cockpit and Portainer)](https://www.youtube.com/watch?v=aLyTWdzDiCg)
- 2018-04-24 - [ElectronicsWizardy: How to setup a Linux Fileserver with Snapraid and Mergerfs Re-Export](https://www.youtube.com/watch?v=D2Klx-X7pFo) * 2018-04-24 - [ElectronicsWizardy: How to setup a Linux Fileserver with Snapraid and Mergerfs Re-Export](https://www.youtube.com/watch?v=D2Klx-X7pFo)
- 2019-03-01 - [Snapraid and Unionfs: Advanced Array Options on Openmediavault (Better than ZFS and Unraid)](https://www.youtube.com/watch?v=FYkdPyCt5FU) * 2019-03-01 - [Snapraid and Unionfs: Advanced Array Options on Openmediavault (Better than ZFS and Unraid)](https://www.youtube.com/watch?v=FYkdPyCt5FU)
- 2019-03-22 - [Gamexplicit: The Perfect Plex Media Server 2019! Part 1 (Hardware)](https://www.youtube.com/watch?v=rJIRPhM2WcE) * 2019-03-22 - [Gamexplicit: The Perfect Plex Media Server 2019! Part 1 (Hardware)](https://www.youtube.com/watch?v=rJIRPhM2WcE)
- 2019-05-13 - [Gamexplicit: The Perfect Plex Media Server 2019! Part 2 (Ubuntu Server 18.04.2 LTS)](https://www.youtube.com/watch?v=aLyTWdzDiCg) * 2019-05-13 - [Gamexplicit: The Perfect Plex Media Server 2019! Part 2 (Ubuntu Server 18.04.2 LTS)](https://www.youtube.com/watch?v=aLyTWdzDiCg)
- 2019-05-20 - [Gamexplicit: The Perfect Plex Media Server 2019! Part 3 (SnapRaid, Samba, Plex)](https://www.youtube.com/watch?v=uW5y43XC-BI) * 2019-05-20 - [Gamexplicit: The Perfect Plex Media Server 2019! Part 3 (SnapRaid, Samba, Plex)](https://www.youtube.com/watch?v=uW5y43XC-BI)
- 2020-08-23 - [Installing OpenMediaVault and SnapRAID and UnionFS (mergerfs)](https://www.youtube.com/watch?v=nDvzXM8UjAI) * 2020-08-23 - [Installing OpenMediaVault and SnapRAID and UnionFS (mergerfs)](https://www.youtube.com/watch?v=nDvzXM8UjAI)
- 2021-06-07 - [I ditched TrueNAS for MergerFS - Chia Plot Storage](https://www.youtube.com/watch?v=tpqFywkbZa4) * 2021-06-07 - [I ditched TrueNAS for MergerFS - Chia Plot Storage](https://www.youtube.com/watch?v=tpqFywkbZa4)
- 2021-08-03 - [Install and configure mergerfs to merge more than one folder in the same place](https://www.youtube.com/watch?v=69zcqEy1674) * 2021-08-03 - [Install and configure mergerfs to merge more than one folder in the same place](https://www.youtube.com/watch?v=69zcqEy1674)
- 2021-08-10 - [Instalando e configurando mergerfs para unir mais de uma pasta no mesmo lugar](https://www.youtube.com/watch?v=-RLxbBNBWhU) * 2021-08-10 - [Instalando e configurando mergerfs para unir mais de uma pasta no mesmo lugar](https://www.youtube.com/watch?v=-RLxbBNBWhU)
- 2021-08-13 - [Let's Convert an Old Laptop to a NAS - What you should do](https://www.youtube.com/watch?v=F1v-TSbOymI) * 2021-08-13 - [Let's Convert an Old Laptop to a NAS - What you should do](https://www.youtube.com/watch?v=F1v-TSbOymI)
- 2021-08-17 - [How to Combine Multiple Disks as One by using MergerFS | Ubuntu 20.04 LTS](https://www.youtube.com/watch?v=9e46pz5Seo4) * 2021-08-17 - [How to Combine Multiple Disks as One by using MergerFS | Ubuntu 20.04 LTS](https://www.youtube.com/watch?v=9e46pz5Seo4)
- 2021-08-20 - [Vamos converter um Laptop antigo para um NAS – O que você deve fazer](https://www.youtube.com/watch?v=q8EK9vWCRTc) * 2021-08-20 - [Vamos converter um Laptop antigo para um NAS – O que você deve fazer](https://www.youtube.com/watch?v=q8EK9vWCRTc)
- 2021-10-29 - [Unlimited space for your Plex using Rclone to connect to your Cloud](https://www.youtube.com/watch?v=ghGconyrF3M) * 2021-10-29 - [Unlimited space for your Plex using Rclone to connect to your Cloud](https://www.youtube.com/watch?v=ghGconyrF3M)
- 2022-12-01 - [Make Your Home Server Go FAST! SSD Caching, 10Gbit Networking, etc.](https://www.youtube.com/watch?v=eRfqC_q3lkM&t=784s) * 2022-12-01 - [Make Your Home Server Go FAST! SSD Caching, 10Gbit Networking, etc.](https://www.youtube.com/watch?v=eRfqC_q3lkM&t=784s)
- 2022-12-07 - [Best RAID for mixed drive sizes. Unraid vs BTRFS vs Snapraid+Mergerfs vs Storage spaces.](https://www.youtube.com/watch?v=NQJkTiLXfgs) * 2022-12-07 - [Best RAID for mixed drive sizes. Unraid vs BTRFS vs Snapraid+Mergerfs vs Storage spaces.](https://www.youtube.com/watch?v=NQJkTiLXfgs)
- 2023-02-21 - [MergerFS + SnapRAID : Forget about RAID 5 in your Home Server !](https://www.youtube.com/watch?v=tX5MA-c6Qq4) * 2023-02-21 - [MergerFS + SnapRAID : Forget about RAID 5 in your Home Server !](https://www.youtube.com/watch?v=tX5MA-c6Qq4)
- 2023-06-26 - [How to install and setup MergerFS](https://www.youtube.com/watch?v=n7piuhTXeG4) * 2023-06-26 - [How to install and setup MergerFS](https://www.youtube.com/watch?v=n7piuhTXeG4)
- 2023-07-31 - [How to recover a dead drive using Snapraid](https://www.youtube.com/watch?v=fmuiRLPcuJE) * 2023-07-31 - [How to recover a dead drive using Snapraid](https://www.youtube.com/watch?v=fmuiRLPcuJE)
- 2024-01-05 - [OpenMediaVault MergerFS Tutorial (Portuguese)](https://www.youtube.com/watch?v=V6Yw86dRUPQ) * 2024-01-05 - [OpenMediaVault MergerFS Tutorial (Portuguese)](https://www.youtube.com/watch?v=V6Yw86dRUPQ)
- 2024-02-19 - [Setup and Install MergerFS and SnapRAID (Part 1)](https://noted.lol/mergerfs-and-snapraid-setup-1/) * 2024-02-19 - [Setup and Install MergerFS and SnapRAID (Part 1)](https://noted.lol/mergerfs-and-snapraid-setup-1/)
- 2024-02-22 - [Setup and Install MergerFS and SnapRAID (Part 2)](https://noted.lol/mergerfs-and-snapraid-setup-part-2/) * 2024-02-22 - [Setup and Install MergerFS and SnapRAID (Part 2)](https://noted.lol/mergerfs-and-snapraid-setup-part-2/)
- 2024-11-15 - [Meu servidor NAS - Parte 18: Recuperando um HD, recuperando o MergerFS e os próximos passos do NAS!](https://www.youtube.com/watch?v=5fy98kPzE3s) * 2024-11-15 - [Meu servidor NAS - Parte 18: Recuperando um HD, recuperando o MergerFS e os próximos passos do NAS!](https://www.youtube.com/watch?v=5fy98kPzE3s)
* 2025-04-23 - [How to build the Perfect Media Server | Part 1 - The Tech Stack | mergerfs, SnapRAID, and docker.](https://www.youtube.com/watch?v=Yt67zz9p0FU)
## Podcasts ## Podcasts
- 2019-11-04 - [Jupiter Extras: A Chat with mergerfs Developer Antonio Musumeci | Jupiter Extras 28](https://www.youtube.com/watch?v=VmJUAyyhSPk) * 2019-11-04 - [Jupiter Extras: A Chat with mergerfs Developer Antonio Musumeci | Jupiter Extras 28](https://www.youtube.com/watch?v=VmJUAyyhSPk)
- 2019-11-07 - [Jupiter Broadcasting: ZFS Isn’t the Only Option | Self-Hosted 5](https://www.youtube.com/watch?v=JEW7UuKhMJ8) * 2019-11-07 - [Jupiter Broadcasting: ZFS Isn’t the Only Option | Self-Hosted 5](https://www.youtube.com/watch?v=JEW7UuKhMJ8)
- 2023-10-08 - [Self Hosted Episode 105 - Sleeper Storage Technology](https://selfhosted.show/105) * 2023-10-08 - [Self Hosted Episode 105 - Sleeper Storage Technology](https://selfhosted.show/105)
## Social Media ## Social Media
- [Reddit](https://www.reddit.com/search/?q=mergerfs&sort=new) * [Reddit](https://www.reddit.com/search/?q=mergerfs&sort=new)
- [X](https://x.com/search?q=mergerfs&src=spelling_expansion_revert_click&f=live) * [X](https://x.com/search?q=mergerfs&src=spelling_expansion_revert_click&f=live)
- [YouTube](https://www.youtube.com/results?search_query=mergerfs&sp=CAI%253D) * [YouTube](https://www.youtube.com/results?search_query=mergerfs&sp=CAI%253D)
- [ServeTheHome Forum](https://forums.servethehome.com/index.php?search/3105813/&q=mergerfs&o=date) * [ServeTheHome Forum](https://forums.servethehome.com/index.php?search/3105813/&q=mergerfs&o=date)

View File

@ -59,6 +59,10 @@ IO.)
## Usage ## Usage
For some suggestions on branch setup see [the page on
branches](config/branches.md#branch-setup).
### Command Line ### Command Line
``` ```

View File

@ -4,14 +4,15 @@
Some storage technologies support what is called "tiered" caching. The Some storage technologies support what is called "tiered" caching. The
placing of smaller, faster storage as a transparent cache to larger, placing of smaller, faster storage as a transparent cache to larger,
slower storage. NVMe, SSD, Optane in front of traditional HDDs for slower storage. NVMe, SSD, or Optane in front of traditional HDDs for
instance. instance.
mergerfs does not natively support any sort of tiered caching. Most mergerfs does not natively support any sort of tiered caching
users have no use for such a feature and its inclusion would currently. The truth is for many users a cache would have little or no
complicate the code as it exists today. However, there are a few advantage over reading and writing directly. They would be
situations where a cache filesystem could help with a typical mergerfs bottlenecked by their network, internet connection, or limited size of
setup. the cache. However, there are a few situations where a tiered cache
setup could help.
1. Fast network, slow filesystems, many readers: You've a 10+Gbps 1. Fast network, slow filesystems, many readers: You've a 10+Gbps
network with many readers and your regular filesystems can't keep network with many readers and your regular filesystems can't keep
@ -34,7 +35,7 @@ dm-cache but there is another solution which requires only mergerfs, a
script to move files around, and a cron job to run said script. script to move files around, and a cron job to run said script.
* Create two mergerfs pools. One which includes just the **slow** * Create two mergerfs pools. One which includes just the **slow**
branches and one which has both the **fast** branches branches and one which has **both** the **fast** branches
(SSD,NVME,etc.) and **slow** branches. The **base** pool and the (SSD,NVME,etc.) and **slow** branches. The **base** pool and the
**cache** pool. **cache** pool.
* The **cache** pool should have the cache branches listed first in * The **cache** pool should have the cache branches listed first in
@ -43,7 +44,7 @@ script to move files around, and a cron job to run said script.
probably be `ff`, `lus`, or `lfs`. The latter two under the probably be `ff`, `lus`, or `lfs`. The latter two under the
assumption that the cache filesystem(s) are far smaller than the assumption that the cache filesystem(s) are far smaller than the
backing filesystems. backing filesystems.
* You can also set the **slow** filesystems mode to `NC` which would * You can also set the **slow** branches' mode to `NC` which would
give you the ability to use other `create` policies though that'd give you the ability to use other `create` policies though that'd
mean if the cache filesystems fill you'd get "out of space" mean if the cache filesystems fill you'd get "out of space"
errors. This however may be good as it would indicate the script errors. This however may be good as it would indicate the script

View File

@ -5,6 +5,9 @@ repo_name: mergerfs
repo_url: https://github.com/trapexit/mergerfs repo_url: https://github.com/trapexit/mergerfs
edit_uri: edit/master/mkdocs/docs/ edit_uri: edit/master/mkdocs/docs/
docs_dir: docs docs_dir: docs
plugins:
- search
- tags
theme: theme:
name: material name: material
logo: logo.png logo: logo.png
@ -62,6 +65,7 @@ nav:
- config/options.md - config/options.md
- config/deprecated_options.md - config/deprecated_options.md
- config/branches.md - config/branches.md
- config/branches_mount_timeout.md
- config/functions_categories_policies.md - config/functions_categories_policies.md
- config/minfreespace.md - config/minfreespace.md
- config/func_readdir.md - config/func_readdir.md

View File

@ -56,6 +56,7 @@ namespace l
{ {
IFERT("async_read"); IFERT("async_read");
IFERT("branches-mount-timeout"); IFERT("branches-mount-timeout");
IFERT("branches-mount-timeout-fail");
IFERT("cache.symlinks"); IFERT("cache.symlinks");
IFERT("cache.writeback"); IFERT("cache.writeback");
IFERT("direct-io-allow-mmap"); IFERT("direct-io-allow-mmap");
@ -85,6 +86,7 @@ Config::Config()
minfreespace(MINFREESPACE_DEFAULT), minfreespace(MINFREESPACE_DEFAULT),
branches(minfreespace), branches(minfreespace),
branches_mount_timeout(0), branches_mount_timeout(0),
branches_mount_timeout_fail(false),
cache_attr(1), cache_attr(1),
cache_entry(1), cache_entry(1),
cache_files(CacheFiles::ENUM::OFF), cache_files(CacheFiles::ENUM::OFF),
@ -139,6 +141,7 @@ Config::Config()
_map["auto_cache"] = &auto_cache; _map["auto_cache"] = &auto_cache;
_map["branches"] = &branches; _map["branches"] = &branches;
_map["branches-mount-timeout"] = &branches_mount_timeout; _map["branches-mount-timeout"] = &branches_mount_timeout;
_map["branches-mount-timeout-fail"] = &branches_mount_timeout_fail;
_map["cache.attr"] = &cache_attr; _map["cache.attr"] = &cache_attr;
_map["cache.entry"] = &cache_entry; _map["cache.entry"] = &cache_entry;
_map["cache.files"] = &cache_files; _map["cache.files"] = &cache_files;

View File

@ -41,6 +41,8 @@
#include "rwlock.hpp" #include "rwlock.hpp"
#include "tofrom_wrapper.hpp" #include "tofrom_wrapper.hpp"
#include "ghc/filesystem.hpp"
#include "fuse.h" #include "fuse.h"
#include <cstdint> #include <cstdint>
@ -55,6 +57,7 @@ typedef ToFromWrapper<bool> ConfigBOOL;
typedef ToFromWrapper<uint64_t> ConfigUINT64; typedef ToFromWrapper<uint64_t> ConfigUINT64;
typedef ToFromWrapper<int> ConfigINT; typedef ToFromWrapper<int> ConfigINT;
typedef ToFromWrapper<std::string> ConfigSTR; typedef ToFromWrapper<std::string> ConfigSTR;
typedef ToFromWrapper<ghc::filesystem::path> ConfigPath;
typedef std::map<std::string,ToFromString*> Str2TFStrMap; typedef std::map<std::string,ToFromString*> Str2TFStrMap;
extern const std::string CONTROLFILE; extern const std::string CONTROLFILE;
@ -108,6 +111,7 @@ public:
ConfigUINT64 minfreespace; ConfigUINT64 minfreespace;
Branches branches; Branches branches;
ConfigUINT64 branches_mount_timeout; ConfigUINT64 branches_mount_timeout;
ConfigBOOL branches_mount_timeout_fail;
ConfigUINT64 cache_attr; ConfigUINT64 cache_attr;
ConfigUINT64 cache_entry; ConfigUINT64 cache_entry;
CacheFiles cache_files; CacheFiles cache_files;
@ -133,7 +137,7 @@ public:
ConfigBOOL link_cow; ConfigBOOL link_cow;
LinkEXDEV link_exdev; LinkEXDEV link_exdev;
LogMetrics log_metrics; LogMetrics log_metrics;
ConfigSTR mountpoint; ConfigPath mountpoint;
MoveOnENOSPC moveonenospc; MoveOnENOSPC moveonenospc;
NFSOpenHack nfsopenhack; NFSOpenHack nfsopenhack;
ConfigBOOL nullrw; ConfigBOOL nullrw;

View File

@ -16,12 +16,11 @@
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/ */
#include "from_string.hpp"
#include "ef.hpp" #include "ef.hpp"
#include "errno.hpp" #include "errno.hpp"
#include <cstdint>
#include <string>
#include <stdlib.h> #include <stdlib.h>
@ -128,4 +127,13 @@ namespace str
{ {
return -EINVAL; return -EINVAL;
} }
int
from(const std::string &value_,
fs::Path *path_)
{
*path_ = value_;
return 0;
}
} }

View File

@ -18,6 +18,8 @@
#pragma once #pragma once
#include "fs_path.hpp"
#include <cstdint> #include <cstdint>
#include <string> #include <string>
@ -29,4 +31,5 @@ namespace str
int from(const std::string &, uint64_t *); int from(const std::string &, uint64_t *);
int from(const std::string &, std::string *); int from(const std::string &, std::string *);
int from(const std::string &, const std::string *); int from(const std::string &, const std::string *);
int from(const std::string &, fs::Path *);
} }

View File

@ -18,7 +18,7 @@
#pragma once #pragma once
#include "ghc/filesystem.hpp" #include "fs_path.hpp"
#include <string> #include <string>
@ -49,8 +49,8 @@ namespace fs
static static
inline inline
int int
mkdir(const ghc::filesystem::path &path_, mkdir(const fs::Path &path_,
const mode_t mode_) const mode_t mode_)
{ {
return fs::mkdir(path_.c_str(),mode_); return fs::mkdir(path_.c_str(),mode_);
} }

View File

@ -30,7 +30,7 @@
#define PAD_LEN 16 #define PAD_LEN 16
#define MAX_ATTEMPTS 3 #define MAX_ATTEMPTS 3
static char const CHARS[] = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"; static char const CHARS[] = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz";
static size_t const CHARS_SIZE = (sizeof(CHARS) - 1); static size_t const CHARS_SIZE = (sizeof(CHARS) - 1);
@ -57,8 +57,8 @@ namespace l
namespace fs namespace fs
{ {
std::tuple<int,std::string> std::tuple<int,std::string>
mktemp_in_dir(std::string const dirpath_, mktemp_in_dir(const std::string dirpath_,
int const flags_) const int flags_)
{ {
int fd; int fd;
int count; int count;
@ -85,10 +85,10 @@ namespace fs
} }
std::tuple<int,std::string> std::tuple<int,std::string>
mktemp(std::string const filepath_, mktemp(const std::string filepath_,
int const flags_) const int flags_)
{ {
ghc::filesystem::path filepath{filepath_}; fs::Path filepath{filepath_};
return fs::mktemp_in_dir(filepath.parent_path(),flags_); return fs::mktemp_in_dir(filepath.parent_path(),flags_);
} }

37
src/fs_mounts.cpp Normal file
View File

@ -0,0 +1,37 @@
#include "fs_mounts.hpp"
#include <cstdio>
#include <mntent.h>
#ifdef __linux__
void
fs::mounts(fs::MountVec &mounts_)
{
FILE *f;
f = setmntent("/proc/mounts","r");
if(f == NULL)
return;
struct mntent *entry;
while((entry = getmntent(f)) != NULL)
{
fs::Mount m;
m.dir = entry->mnt_dir;
m.fsname = entry->mnt_fsname;
m.type = entry->mnt_type;
m.opts = entry->mnt_opts;
mounts_.emplace_back(std::move(m));
}
endmntent(f);
}
#else
void
fs::mounts(fs::MountVec &mounts_)
{
}
#endif

21
src/fs_mounts.hpp Normal file
View File

@ -0,0 +1,21 @@
#pragma once
#include "fs_path.hpp"
#include <vector>
namespace fs
{
struct Mount
{
fs::Path dir;
std::string fsname;
std::string type;
std::string opts;
};
typedef std::vector<fs::Mount> MountVec;
void mounts(fs::MountVec &mounts);
}

View File

@ -19,6 +19,10 @@
#include "fs_wait_for_mount.hpp" #include "fs_wait_for_mount.hpp"
#include "syslog.hpp" #include "syslog.hpp"
#include "fs_exists.hpp"
#include "fs_lstat.hpp"
#include "fs_lgetxattr.hpp"
#include <functional> #include <functional>
#include <thread> #include <thread>
#include <unordered_set> #include <unordered_set>
@ -44,6 +48,40 @@ namespace std
}; };
} }
static
bool
_branch_is_mounted(const struct stat &src_st_,
const fs::Path &branch_path_)
{
int rv;
struct stat st;
fs::Path filepath;
rv = fs::lgetxattr(branch_path_,"user.mergerfs.branch",NULL,0);
if(rv != -1)
return true;
filepath = branch_path_ / ".mergerfs.branch";
rv = fs::exists(filepath);
if(rv)
return true;
rv = fs::lgetxattr(branch_path_,"user.mergerfs.branch_mounts_here",NULL,0);
if(rv != -1)
return false;
filepath = branch_path_ / ".mergerfs.branch_mounts_here";
rv = fs::exists(filepath);
if(rv)
return false;
rv = fs::lstat(branch_path_,&st);
if(rv == 0)
return (st.st_dev != src_st_.st_dev);
return false;
}
static static
void void
_check_mounted(const struct stat &src_st_, _check_mounted(const struct stat &src_st_,
@ -56,30 +94,23 @@ _check_mounted(const struct stat &src_st_,
for(auto const &tgt_path : tgt_paths_) for(auto const &tgt_path : tgt_paths_)
{ {
int rv; bool mounted;
struct stat tgt_st;
rv = fs::stat(tgt_path,&tgt_st); mounted = ::_branch_is_mounted(src_st_,tgt_path);
if(rv == 0) if(mounted)
{ successes.push_back(tgt_path);
if(tgt_st.st_dev != src_st_.st_dev)
successes.push_back(tgt_path);
else
failures.push_back(tgt_path);
}
else else
{ failures.push_back(tgt_path);
failures.push_back(tgt_path);
}
} }
} }
static static
void int
_wait_for_mount(const struct stat &src_st_, _wait_for_mount(const struct stat &src_st_,
const fs::PathVector &tgt_paths_, const fs::PathVector &tgt_paths_,
const std::chrono::milliseconds &timeout_) const std::chrono::milliseconds &timeout_)
{ {
bool first_loop;
fs::PathVector successes; fs::PathVector successes;
fs::PathVector failures; fs::PathVector failures;
std::unordered_set<fs::Path> tgt_paths; std::unordered_set<fs::Path> tgt_paths;
@ -90,6 +121,7 @@ _wait_for_mount(const struct stat &src_st_,
now = std::chrono::steady_clock::now(); now = std::chrono::steady_clock::now();
deadline = now + timeout_; deadline = now + timeout_;
first_loop = true;
while(true) while(true)
{ {
if(tgt_paths.empty()) if(tgt_paths.empty())
@ -100,10 +132,17 @@ _wait_for_mount(const struct stat &src_st_,
successes.clear(); successes.clear();
failures.clear(); failures.clear();
::_check_mounted(src_st_,tgt_paths,&successes,&failures); ::_check_mounted(src_st_,tgt_paths,&successes,&failures);
for(auto const &path : successes) for(const auto &path : successes)
{ {
tgt_paths.erase(path); tgt_paths.erase(path);
syslog_info("%s is mounted",path.string().c_str()); syslog_info("%s is mounted",path.c_str());
}
if(first_loop)
{
for(const auto &path : failures)
syslog_notice("%s is not mounted, waiting",path.c_str());
first_loop = false;
} }
std::this_thread::sleep_for(SLEEP_DURATION); std::this_thread::sleep_for(SLEEP_DURATION);
@ -111,26 +150,24 @@ _wait_for_mount(const struct stat &src_st_,
} }
for(auto const &path : failures) for(auto const &path : failures)
syslog_notice("%s not mounted within timeout",path.string().c_str()); syslog_notice("%s not mounted within timeout",path.c_str());
if(!failures.empty())
syslog_warning("Continuing to mount mergerfs despite %u branches not " return failures.size();
"being different from the mountpoint filesystem",
failures.size());
} }
void int
fs::wait_for_mount(const fs::Path &src_path_, fs::wait_for_mount(const fs::Path &src_path_,
const fs::PathVector &tgt_paths_, const fs::PathVector &tgt_paths_,
const std::chrono::milliseconds &timeout_) const std::chrono::milliseconds &timeout_)
{ {
int rv; int rv;
struct stat src_st; struct stat src_st = {0};
rv = fs::stat(src_path_,&src_st); rv = fs::stat(src_path_,&src_st);
if(rv == -1) if(rv == -1)
return syslog_error("Error stat'ing mount path: %s (%s)", syslog_error("Error stat'ing mount path: %s (%s)",
src_path_.c_str(), src_path_.c_str(),
strerror(errno)); strerror(errno));
::_wait_for_mount(src_st,tgt_paths_,timeout_); return ::_wait_for_mount(src_st,tgt_paths_,timeout_);
} }

View File

@ -1,6 +1,5 @@
/* /*
ISC License ISC License
Copyright (c) 2023, Antonio SJ Musumeci <trapexit@spawn.link> Copyright (c) 2023, Antonio SJ Musumeci <trapexit@spawn.link>
Permission to use, copy, modify, and/or distribute this software for any Permission to use, copy, modify, and/or distribute this software for any
@ -29,7 +28,7 @@
namespace fs namespace fs
{ {
void int
wait_for_mount(const fs::Path &srcpath, wait_for_mount(const fs::Path &srcpath,
const fs::PathVector &tgtpaths, const fs::PathVector &tgtpaths,
const std::chrono::milliseconds &timeout); const std::chrono::milliseconds &timeout);

View File

@ -19,12 +19,13 @@
#include "fs_readahead.hpp" #include "fs_readahead.hpp"
#include "syslog.hpp" #include "syslog.hpp"
#include "fs_path.hpp"
#include "fs_exists.hpp"
#include "fmt/core.h" #include "fmt/core.h"
#include "fuse.h" #include "fuse.h"
#include "ghc/filesystem.hpp"
#include <thread> #include <thread>
#include <algorithm> #include <algorithm>
@ -82,7 +83,7 @@ namespace l
std::fstream f; std::fstream f;
uint64_t max_pages_limit; uint64_t max_pages_limit;
if(ghc::filesystem::exists(MAX_PAGES_LIMIT_FILEPATH)) if(fs::exists(MAX_PAGES_LIMIT_FILEPATH))
{ {
if(cfg_->fuse_msg_size > MAX_FUSE_MSG_SIZE) if(cfg_->fuse_msg_size > MAX_FUSE_MSG_SIZE)
syslog_info("fuse_msg_size > %u: setting it to %u", syslog_info("fuse_msg_size > %u: setting it to %u",
@ -141,8 +142,8 @@ namespace l
static static
void void
readahead(const std::string path_, readahead(const fs::Path path_,
const int readahead_) const int readahead_)
{ {
int rv; int rv;

View File

@ -32,7 +32,6 @@
using std::string; using std::string;
using std::vector; using std::vector;
namespace gfs = ghc::filesystem;
namespace error namespace error
{ {
@ -230,8 +229,8 @@ namespace l
fuse_timeouts_t *timeouts_) fuse_timeouts_t *timeouts_)
{ {
int rv; int rv;
gfs::path target(oldpath_); fs::Path target(oldpath_);
gfs::path linkpath(newpath_); fs::Path linkpath(newpath_);
target = target.lexically_relative(linkpath.parent_path()); target = target.lexically_relative(linkpath.parent_path());
@ -278,11 +277,11 @@ namespace l
static static
int int
link_exdev_abs_pool_symlink(const std::string &mount_, link_exdev_abs_pool_symlink(const fs::Path mount_,
const char *oldpath_, const char *oldpath_,
const char *newpath_, const char *newpath_,
struct stat *st_, struct stat *st_,
fuse_timeouts_t *timeouts_) fuse_timeouts_t *timeouts_)
{ {
int rv; int rv;
StrVec basepaths; StrVec basepaths;

View File

@ -315,7 +315,7 @@ namespace l
int int
rename_exdev_abs_symlink(const Policy::Action &actionPolicy_, rename_exdev_abs_symlink(const Policy::Action &actionPolicy_,
const Branches::CPtr &branches_, const Branches::CPtr &branches_,
const std::string &mount_, const gfs::path &mount_,
const gfs::path &oldfusepath_, const gfs::path &oldfusepath_,
const gfs::path &newfusepath_) const gfs::path &newfusepath_)
{ {

View File

@ -165,26 +165,51 @@ namespace l
} }
static static
void bool
wait_for_mount(const Config::Read &cfg_) wait_for_mount(const Config::Read &cfg_)
{ {
int failures;
fs::PathVector paths; fs::PathVector paths;
std::chrono::milliseconds timeout; std::chrono::milliseconds timeout;
paths = cfg_->branches->to_paths(); paths = cfg_->branches->to_paths();
syslog_info("Waiting %u seconds for branches to mount", syslog_info("Waiting %u seconds for %zu branches to mount",
(uint64_t)cfg_->branches_mount_timeout); (uint64_t)cfg_->branches_mount_timeout,
paths.size());
timeout = std::chrono::milliseconds(cfg_->branches_mount_timeout * 1000); timeout = std::chrono::milliseconds(cfg_->branches_mount_timeout * 1000);
fs::wait_for_mount((std::string)cfg_->mountpoint, failures = fs::wait_for_mount(cfg_->mountpoint,
paths, paths,
timeout); timeout);
if(failures)
{
if(cfg_->branches_mount_timeout_fail)
{
syslog_error("%d of %zu branches were not mounted"
" within the timeout of %zus. Exiting",
failures,
paths.size(),
(uint64_t)cfg_->branches_mount_timeout);
return true;
}
syslog_warning("Continuing to mount mergerfs despite %d branches not "
"being different from the mountpoint filesystem",
failures);
}
else
{
syslog_info("All %zd branches are mounted",
paths.size());
}
return false;
} }
static static
void void
lazy_umount(const std::string target_) lazy_umount(const fs::Path target_)
{ {
int rv; int rv;
@ -276,7 +301,13 @@ namespace l
} }
if(cfg->branches_mount_timeout > 0) if(cfg->branches_mount_timeout > 0)
l::wait_for_mount(cfg); {
bool failure;
failure = l::wait_for_mount(cfg);
if(failure)
return 1;
}
l::setup_resources(cfg->scheduling_priority); l::setup_resources(cfg->scheduling_priority);
l::setup_signal_handlers(); l::setup_signal_handlers();

View File

@ -421,7 +421,7 @@ check_for_mount_loop(Config::Write &cfg_,
fs::PathVector branches; fs::PathVector branches;
std::error_code ec; std::error_code ec;
mount = (std::string)cfg_->mountpoint; mount = *cfg_->mountpoint;
branches = cfg_->branches->to_paths(); branches = cfg_->branches->to_paths();
for(const auto &branch : branches) for(const auto &branch : branches)
{ {

View File

@ -16,6 +16,8 @@
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/ */
#include "to_string.hpp"
#include <cstdint> #include <cstdint>
#include <string> #include <string>
@ -55,4 +57,10 @@ namespace str
{ {
return s_; return s_;
} }
std::string
to(const fs::Path &path_)
{
return path_.string();
}
} }

View File

@ -18,6 +18,8 @@
#pragma once #pragma once
#include "fs_path.hpp"
#include <cstdint> #include <cstdint>
#include <string> #include <string>
@ -28,4 +30,5 @@ namespace str
std::string to(const int); std::string to(const int);
std::string to(const uint64_t); std::string to(const uint64_t);
std::string to(const std::string&); std::string to(const std::string&);
std::string to(const fs::Path&);
} }

View File

@ -64,6 +64,13 @@ public:
return _data; return _data;
} }
const
T&
operator*() const
{
return _data;
}
T* T*
operator->() operator->()
{ {