mirror of
https://github.com/trapexit/mergerfs.git
synced 2025-04-25 14:34:04 +08:00
Merge pull request #1447 from trapexit/branchcfg
Improve mount waiting + misc doc improvements
This commit is contained in:
commit
8ee998e8aa
@ -1,18 +1,20 @@
|
||||
# branches
|
||||
|
||||
The 'branches' argument is a colon (':') delimited list of paths to be
|
||||
The `branches` argument is a colon (':') delimited list of paths to be
|
||||
pooled together. It does not matter if the paths are on the same or
|
||||
different filesystems nor does it matter the filesystem type (within
|
||||
reason). Used and available space will not be duplicated for paths on
|
||||
the same filesystem and any features which aren't supported by the
|
||||
underlying filesystem (such as file attributes or extended attributes)
|
||||
will return the appropriate errors.
|
||||
different filesystems nor does it matter [the filesystem
|
||||
type](../faq/compatibility_and_integration.md/#what-filesystems-can-be-used-as-branches)
|
||||
(within reason). Used and available space metrics will not be
|
||||
duplicated for paths on the same filesystem and any features which
|
||||
aren't supported by the underlying filesystem (such as file attributes
|
||||
or extended attributes) will return the appropriate errors.
|
||||
|
||||
Branches currently have two options which can be set. A type which
|
||||
impacts whether or not the branch is included in a policy calculation
|
||||
and a individual minfreespace value. The values are set by prepending
|
||||
an `=` at the end of a branch designation and using commas as
|
||||
delimiters. Example: `/mnt/drive=RW,1234`
|
||||
Branches currently have two options which can be set. A
|
||||
[mode](#branch-mode) which impacts whether or not the branch is
|
||||
included in a policy calculation and a individual
|
||||
[minfreespace](#minfreespace) value. The values are set by
|
||||
prepending an `=` at the end of a branch designation and using commas
|
||||
as delimiters. Example: `/mnt/drive=RW,1234`
|
||||
|
||||
|
||||
### branch mode
|
||||
@ -29,7 +31,7 @@ delimiters. Example: `/mnt/drive=RW,1234`
|
||||
### minfreespace
|
||||
|
||||
Same purpose and syntax as the [global option](minfreespace.md) but
|
||||
specific to the branch. If not set the global value is used.
|
||||
specific to the branch. Defaults to the global value.
|
||||
|
||||
|
||||
### globbing
|
||||
@ -43,8 +45,8 @@ apply the glob itself.**
|
||||
# mergerfs /mnt/hdd\*:/mnt/ssd /media
|
||||
```
|
||||
|
||||
The above line will use all mount points in /mnt prefixed with **hdd**
|
||||
and **ssd**.
|
||||
The above line will use all directories in /mnt prefixed with **hdd**
|
||||
as well as **ssd**.
|
||||
|
||||
To have the pool mounted at boot or otherwise accessible from related
|
||||
tools use `/etc/fstab`.
|
||||
@ -54,4 +56,154 @@ tools use `/etc/fstab`.
|
||||
/mnt/hdd*:/mnt/ssd /media mergerfs minfreespace=16G 0 0
|
||||
```
|
||||
|
||||
**NOTE:** The globbing is done at mount or when updated using the runtime API. If a new directory is added matching the glob after the fact it will not be automatically included.
|
||||
**NOTE:** The globbing is done at mount or when updated using the
|
||||
runtime API. If a new directory is added matching the glob after the
|
||||
fact it will not be automatically included.
|
||||
|
||||
A convenient way to configure branches is to use
|
||||
[symlinks](../quickstart.md/#etcfstab-w-config-file).
|
||||
|
||||
|
||||
## branch setup
|
||||
|
||||
mergerfs does not require any special setup of branch paths in order
|
||||
to be used however here are some suggestions.
|
||||
|
||||
|
||||
### layout
|
||||
|
||||
When you have a large collection of storage devices, types of storage,
|
||||
and types of interconnects it can be useful to have a verbose naming
|
||||
convention. My preference is to create a directory within `/mnt/` for
|
||||
each major type of storage device and/or interconnect. `hdd` for
|
||||
traditional hard drives; `ssd` for slower SATA based SSDs; `nvme` for
|
||||
M.2 or U.2 NVME SSDs; `remote` for NFS, SMB, sshfs, etc.
|
||||
|
||||
The mount points within those directories are named after the size of
|
||||
the storage and its serial number in the case of physical storage and
|
||||
the hostname and remote filesystem for remote.
|
||||
|
||||
```
|
||||
$ ls -lh /mnt/
|
||||
total 16K
|
||||
drwxr-xr-x 8 root root 4.0K Aug 18 2024 hdd
|
||||
drwxr-xr-x 6 root root 4.0K Oct 8 2024 nvme
|
||||
drwxr-xr-x 3 root root 4.0K Aug 24 2024 remote
|
||||
drwxr-xr-x 3 root root 4.0K Jul 14 2024 ssd
|
||||
|
||||
$ ls -lh /mnt/hdd/
|
||||
total 8K
|
||||
d--------- 2 root root 4.0K Apr 14 15:58 10T-01234567
|
||||
d--------- 2 root root 4.0K Apr 12 20:51 20T-12345678
|
||||
|
||||
$ ls -lh /mnt/nvme/
|
||||
total 8K
|
||||
d--------- 2 root root 4.0K Apr 14 16:00 1T-ABCDEFGH
|
||||
d--------- 2 root root 4.0K Apr 14 23:24 1T-BCDEFGHI
|
||||
|
||||
$ ls -lh /mnt/remote/
|
||||
total 8K
|
||||
d--------- 2 root root 4.0K Apr 12 20:23 foo-sshfs
|
||||
d--------- 2 root root 4.0K Apr 12 20:24 bar-nfs
|
||||
|
||||
# You can find the serial number of a drive using lsblk
|
||||
$ lsblk -d -o NAME,PATH,SIZE,SERIAL
|
||||
NAME PATH SIZE SERIAL
|
||||
sda /dev/sda 9.1T 01234567
|
||||
sdb /dev/sdb 18.2T 12345678
|
||||
nvme0n1 /dev/nvme0n1 953.9G ABCDEFGH
|
||||
nvme1n1 /dev/nvme1n1 953.9G BCDEFGHI
|
||||
```
|
||||
|
||||
|
||||
### permissions and mode
|
||||
|
||||
#### mount points
|
||||
|
||||
To ensure the directory is **only** used as a point to mount another
|
||||
filesystem it is good to lock it down as much as possible. Be sure to
|
||||
do this **before** mounting a filesystem to it.
|
||||
|
||||
```
|
||||
$ sudo chown root:root /mnt/hdd/10T-XYZ
|
||||
$ sudo chmod 0000 /mnt/hdd/10T-XYZ
|
||||
$ sudo setfattr -n user.mergerfs.branch_mounts_here
|
||||
$ sudo chattr +i /mnt/hdd/10T-XYZ
|
||||
```
|
||||
|
||||
The extended attribute `user.mergerfs.branch_mounts_here` is used by
|
||||
the [branches-mount-timeout](branches_mount_timeout.md) option to
|
||||
recognize whether or not a mergerfs branch path points to the intended
|
||||
filesystem.
|
||||
|
||||
The `chattr` is likely to only work on EXT{2,3,4} filesystems but will
|
||||
restrict even `root` from modifying the directory or its content.
|
||||
|
||||
|
||||
#### mounted filesystems
|
||||
|
||||
For those new to Linux, intending to be the primary individual logged
|
||||
into the system, or simply want to simplify permissions it is
|
||||
recommended to set the root of mounted filesystems like `/tmp/` is set
|
||||
to. Owned by `root`, `ugo+rwx` and [sticky
|
||||
bit](https://en.wikipedia.org/wiki/Sticky_bit) set.
|
||||
|
||||
This must be done **after** mounting the filesystem to the target
|
||||
mount point.
|
||||
|
||||
```
|
||||
$ sudo chown root:root /mnt/hdd/10T-SERIALNUM
|
||||
$ sudo chmod 1777 /mnt/hdd/10T-SERIALNUM
|
||||
$ sudo setfattr -n user.mergerfs.branch /mnt/hdd/10T-SERIALNUM
|
||||
```
|
||||
|
||||
|
||||
### formatting
|
||||
|
||||
While even less relevant to mergerfs than the details above the topic
|
||||
does com up regularly with mergerfs users. When it comes to
|
||||
partitioning and formatting it is suggested to keep things
|
||||
simple. Most users of mergerfs will be using the whole drive capacity
|
||||
and as such have a singular partition and filesystem on that
|
||||
partition. Something many people don't realize is that the partition
|
||||
is not necessary and actually can become problematic.
|
||||
|
||||
Rather than creating paritions just format the block device. Let's use
|
||||
`/dev/sda` as an example.
|
||||
|
||||
```
|
||||
$ lsblk -d -o NAME,PATH,SERIAL /dev/sda
|
||||
NAME PATH SERIAL
|
||||
sda /dev/sda 01234567
|
||||
|
||||
$ sudo mkfs.ext4 -m0 -L 01234567 /dev/sda
|
||||
mke2fs 1.47.0 (5-Feb-2023)
|
||||
Discarding device blocks: done
|
||||
Creating filesystem with 262144 4k blocks and 65536 inodes
|
||||
Filesystem UUID: ede8bf96-9aff-464e-a4fb-a4ab8a197cd7
|
||||
Superblock backups stored on blocks:
|
||||
32768, 98304, 163840, 229376
|
||||
|
||||
Allocating group tables: done
|
||||
Writing inode tables: done
|
||||
Creating journal (8192 blocks): done
|
||||
Writing superblocks and filesystem accounting information: done
|
||||
```
|
||||
|
||||
You can then use the serial number as the identifier in `fstab`.
|
||||
|
||||
```
|
||||
# <file system> <mount point> <type> <options> <dump> <pass>
|
||||
LABEL=01234567 /mnt/hdd/10TB-01234567 auto nofail 0 2
|
||||
```
|
||||
|
||||
Benefits of doing it this way?
|
||||
|
||||
* One less thing (partitions) to configure and worry about.
|
||||
* Guarantees alignment of blocks
|
||||
* Makes it easier to use the drive with different enclosures. Some
|
||||
SATA to USB adapters use 4K blocks while the drive itself is using
|
||||
512b. If you create partitions with one block size and move the
|
||||
drive to a controller that uses the other the offset on the device
|
||||
where the filesystem starts will be misinterpreted. It is possible
|
||||
to manually fix this but it isn't well documented and avoidable.
|
||||
|
48
mkdocs/docs/config/branches_mount_timeout.md
Normal file
48
mkdocs/docs/config/branches_mount_timeout.md
Normal file
@ -0,0 +1,48 @@
|
||||
# branches-mount-timeout
|
||||
|
||||
Default: `0`
|
||||
|
||||
mergerfs is compatible with any path, whether it be a mount point or a
|
||||
directory on your [root
|
||||
filesystem.](https://en.wikipedia.org/wiki/Root_directory) It doesn’t
|
||||
require branch paths to be mounted or to match a specific filesystem
|
||||
at startup, and it operates effectively without needing details about
|
||||
the intended filesystem. This flexibility eliminates the need to
|
||||
manage mount ordering, which is particularly advantageous on modern
|
||||
systems where filesystems are mounted asynchronously, resulting in
|
||||
unpredictable mount sequences. While
|
||||
[systemd](https://www.freedesktop.org/software/systemd/man/latest/systemd.mount.html)
|
||||
offers a way to enforce mount dependencies using the
|
||||
[x-systemd.requires=PATH](https://www.freedesktop.org/software/systemd/man/latest/systemd.mount.html#x-systemd.requires=)
|
||||
option in /etc/fstab, applying this to every branch path is both
|
||||
cumbersome and susceptible to errors.
|
||||
|
||||
`branches-mount-timeout` will cause mergerfs to wait for all branches
|
||||
to become "mounted." The logic to determine "mounted" is as follows.
|
||||
|
||||
1. It is mounted if the branch directory has the extended attribute
|
||||
`user.mergerfs.branch`
|
||||
2. It is mounted if the branch directory contains a file named
|
||||
`.mergerfs.branch`
|
||||
3. It is **not** mounted if the branch directory has the extended
|
||||
attribute `user.mergerfs.branch_mounts_here`
|
||||
4. It is **not** mounted if the branch directory contains
|
||||
a file named `.mergerfs.branch_mounts_here`
|
||||
5. It is mounted if the `st_dev` value of the mergerfs mountpoint is
|
||||
different from the branch path.
|
||||
|
||||
**NOTE:** If on a `systemd` based system and using `fstab` it is a
|
||||
good idea to set the mount option
|
||||
[x-systemd.mount-timeout](https://www.freedesktop.org/software/systemd/man/latest/systemd.mount.html#x-systemd.mount-timeout=)
|
||||
to some value longer than `branches-mount-timeout`.
|
||||
|
||||
|
||||
## branches-mount-timeout-fail
|
||||
|
||||
Default: `false`
|
||||
|
||||
When set to `true` mergerfs will fail entirely after
|
||||
`branches-mount-timeout` expires without all branches being
|
||||
mounted. If set to `false` it will simply ignore the mount status of
|
||||
the branches and continue on. The details will be
|
||||
[logged](../error_handling_and_logging.md).
|
@ -1,28 +1,16 @@
|
||||
# functions, categories and policies
|
||||
|
||||
The POSIX filesystem API is made up of a number of
|
||||
functions. **creat**, **stat**, **chown**, etc. For ease of
|
||||
configuration in mergerfs, most of the core functions are grouped into
|
||||
3 categories: **action**, **create**, and **search**. These functions
|
||||
and categories can be assigned a policy which dictates which branch is
|
||||
chosen when performing that function.
|
||||
The POSIX filesystem API is made up of a number of functions. `creat`,
|
||||
`stat`, `chown`, etc. For ease of configuration in mergerfs, most of
|
||||
the core functions are grouped into 3 categories: `action`, `create`,
|
||||
and `search`. These functions and categories can be assigned a policy
|
||||
which dictates which branch is chosen when performing that
|
||||
function.
|
||||
|
||||
Some functions, listed in the category `N/A` below, can not be
|
||||
assigned the normal policies because they are directly related to a
|
||||
file which has already been opened.
|
||||
|
||||
When using policies which are based on a branch's available space the
|
||||
base path provided is used. Not the full path to the file in
|
||||
question. Meaning that mounts in the branch won't be considered in the
|
||||
space calculations. The reason is that it doesn't really work for
|
||||
non-path preserving policies and can lead to non-obvious behaviors.
|
||||
|
||||
NOTE: While any policy can be assigned to a function or category,
|
||||
some may not be very useful in practice. For instance: **rand**
|
||||
(random) may be useful for file creation (create) but could lead to
|
||||
very odd behavior if used for `chmod` if there were more than one copy
|
||||
of the file.
|
||||
|
||||
|
||||
## Functions and their Category classifications
|
||||
|
||||
@ -34,43 +22,61 @@ of the file.
|
||||
| N/A | fchmod, fchown, futimens, ftruncate, fallocate, fgetattr, fsync, ioctl (files), read, readdir, release, statfs, write, copy_file_range |
|
||||
|
||||
In cases where something may be searched for (such as a path to clone)
|
||||
**getattr** will usually be used.
|
||||
`getattr` will usually be used.
|
||||
|
||||
|
||||
## Policies
|
||||
|
||||
See below for [available policies and their descriptions](#policy-descriptions).
|
||||
|
||||
A policy is an algorithm designed to select one or more branches for a
|
||||
function to operate on.
|
||||
|
||||
Any function in the `create` category will clone the relative path if
|
||||
needed. Some other functions (`rename`,`link`,`ioctl`) have special
|
||||
requirements or behaviors which you can read more about below.
|
||||
Policies do not actually manage the filesystem or layout. They are
|
||||
strictly responsible for deciding which files or branches will be
|
||||
worked on in relation to the function being performed. Once the branch
|
||||
is chosen other parts of the system does what is necessary to
|
||||
accomplish the function. Such as the cloning of directories between
|
||||
branches.
|
||||
|
||||
When using policies which are based on a branch's available space the
|
||||
branch base path provided is used. Not the full path to the file or
|
||||
directory in question. Meaning that mounts within the branch will not
|
||||
be considered in the space calculations.
|
||||
|
||||
NOTE: While any policy can be assigned to a function or category, some
|
||||
may not be very useful in practice. For instance: `rand` (random) may
|
||||
be useful for file creation but could lead to very odd behavior if
|
||||
used for `chmod` if there were more than one copy of the file. Unless
|
||||
users find this flexibility useful it will likely be removed in the
|
||||
future.
|
||||
|
||||
|
||||
## Filtering
|
||||
|
||||
Most policies basically search branches and create a list of files / paths
|
||||
for functions to work on. The policy is responsible for filtering and
|
||||
sorting the branches. Filters include **minfreespace**, whether or not
|
||||
a branch is mounted read-only, and the branch tagging
|
||||
Most policies search branches and create a list of files / paths for
|
||||
functions to work on. The policy is responsible for filtering and
|
||||
sorting the branches. Filters include [minfreespace](minfreespace.md),
|
||||
whether or not a branch is mounted read-only, and the branch mode
|
||||
(RO,NC,RW). These filters are applied across most policies.
|
||||
|
||||
- No **search** function policies filter.
|
||||
- All **action** function policies filter out branches which are
|
||||
mounted **read-only** or tagged as **RO (read-only)**.
|
||||
- All **create** function policies filter out branches which are
|
||||
mounted **read-only**, tagged **RO (read-only)** or **NC (no
|
||||
create)**, or has available space less than `minfreespace`.
|
||||
- No `search` function policies filter.
|
||||
- All `action` function policies filter out branches which are
|
||||
mounted **read-only** or mode is **RO (read-only)**.
|
||||
- All `create` function policies filter out branches which are
|
||||
mounted **read-only**, mode **RO (read-only)** or **NC (no
|
||||
create)**, or has available space less than
|
||||
[minfreespace](minfreespace.md).
|
||||
|
||||
Policies may have their own additional filtering such as those that
|
||||
require existing paths to be present.
|
||||
|
||||
If all branches are filtered an error will be returned. Typically
|
||||
**EROFS** (read-only filesystem) or **ENOSPC** (no space left on
|
||||
`EROFS` (read-only filesystem) or `ENOSPC` (no space left on
|
||||
device) depending on the most recent reason for filtering a
|
||||
branch. **ENOENT** will be returned if no eligible branch is found.
|
||||
branch.
|
||||
|
||||
If **create**, **mkdir**, **mknod**, or **symlink** fail with `EROFS`
|
||||
If `create`, `mkdir`, `mknod`, or `symlink` fail with `EROFS`
|
||||
or other fundamental errors then mergerfs will mark any branch found
|
||||
to be read-only as such (IE will set the mode `RO`) and will rerun the
|
||||
policy and try again. This is mostly for `ext4` filesystems that can
|
||||
@ -82,16 +88,12 @@ suddenly become read-only when it encounters an error.
|
||||
Policies, as described below, are of two basic classifications. `path
|
||||
preserving` and `non-path preserving`.
|
||||
|
||||
All policies which start with `ep` (**epff**, **eplfs**, **eplus**,
|
||||
**epmfs**, **eprand**) are `path preserving`. `ep` stands for
|
||||
`existing path`.
|
||||
All policies which start with `ep` (`epff`, `eplfs`, `eplus`, `epmfs`,
|
||||
`eprand`) are `path preserving`. `ep` stands for `existing path`.
|
||||
|
||||
A path preserving policy will only consider branches where the relative
|
||||
path being accessed already exists.
|
||||
|
||||
When using non-path preserving policies paths will be cloned to target
|
||||
branches as necessary.
|
||||
|
||||
With the `msp` or `most shared path` policies they are defined as
|
||||
`path preserving` for the purpose of controlling `link` and `rename`'s
|
||||
behaviors since `ignorepponrename` is available to disable that
|
||||
@ -141,5 +143,5 @@ policies is not appropriate.
|
||||
| Category | Policy |
|
||||
| -------- | ------ |
|
||||
| action | epall |
|
||||
| create | epmfs |
|
||||
| create | pfrd |
|
||||
| search | ff |
|
||||
|
@ -17,31 +17,30 @@ anywhere.
|
||||
## Why are all my files ending up on 1 filesystem?!
|
||||
|
||||
Did you start with empty filesystems? Are you using an `existing path`
|
||||
policy?
|
||||
policy such as `epmfs`?
|
||||
|
||||
The default create policy is `epmfs`. That is a path preserving
|
||||
algorithm. With such a policy for `mkdir` and `create` with a set of
|
||||
empty filesystems it will select only 1 filesystem when the first
|
||||
directory is created. Anything, files or directories, created in that
|
||||
directory will be placed on the same branch because it is preserving
|
||||
paths. That is the expected behavior.
|
||||
If starting with a set of empty filesystems such a policy will select
|
||||
only 1 filesystem when the first directory is created. Anything
|
||||
created in that directory will be placed on the same branch because
|
||||
that is the point of such a policy. That is the expected behavior.
|
||||
|
||||
This may catch new users off guard but this policy is the safest
|
||||
policy to have as default as it will not change the general layout of
|
||||
the underlying branches. If you do not care about path preservation
|
||||
([most should
|
||||
not](configuration_and_policies.md#how-can-i-ensure-files-are-collocated-on-the-same-branch))
|
||||
and wish your files to be spread across all your filesystems change
|
||||
`create.category` to `pfrd`, `rand`, `mfs` or a similarly non-path
|
||||
restrictive [policy](../config/functions_categories_policies.md).
|
||||
This may catch users off guard but the point of those `existing path`
|
||||
policies is to preserve the directory layout and you need to consider
|
||||
what that means one event at a time.
|
||||
|
||||
To "fix" this situation change the policy. Most users [have little
|
||||
reason](configuration_and_policies.md#how-can-i-ensure-files-are-collocated-on-the-same-branch)
|
||||
to use such a policy. See the [quick start guide](../quickstart.md)
|
||||
and [FAQ](configuration_and_policies.md) for policy recommendations.
|
||||
|
||||
|
||||
## Why do I get an "out of space" / "no space left on device" / ENOSPC error even though there appears to be lots of space available?
|
||||
|
||||
First make sure you've read the sections above about [policies, path
|
||||
First make sure you've read the sections about [policies, path
|
||||
preservation, branch
|
||||
filtering,](../config/functions_categories_policies.md) and the
|
||||
options `minfreespace`, [moveonenospc](../config/moveonenospc.md),
|
||||
options [minfreespace](../config/minfreespace.md),
|
||||
[moveonenospc](../config/moveonenospc.md),
|
||||
[statfs](../config/statfs.md), and
|
||||
[statfs_ignore](../config/statfs.md#statfs_ignore).
|
||||
|
||||
|
@ -2,79 +2,79 @@
|
||||
|
||||
## Tutorials / Articles
|
||||
|
||||
- 2016-02-02 - [Linuxserver.io: The Perfect Media Server 2016](https://blog.linuxserver.io/2016/02/02/the-perfect-media-server-2016/)
|
||||
- 2016-08-31 - [ZackReed.me: Mergerfs – another good option to pool your SnapRAID disks](https://zackreed.me/mergerfs-another-good-option-to-pool-your-snapraid-disks/)
|
||||
- 2016-11-06 - [Linuxserver.io: Revisiting the HP ProLiant Gen8 G1610T Microserver](https://blog.linuxserver.io/2016/11/06/revisiting-the-hp-proliant-gen8-g1610t-microserver/)
|
||||
- 2017-01-17 - [Setting up mergerfs on JBOD's (or a poor mans storage array)](http://corywestropp.com/develop/articles/setting-up-mergerfs/)
|
||||
- 2017-06-24 - [Linuxserver.io: The Perfect Media Server 2017](https://blog.linuxserver.io/2017/06/24/the-perfect-media-server-2017/)
|
||||
- 2018-02-19 - [Teknophiles: Disk Pooling in Linux with mergerFS](https://web.archive.org/web/20210324184857/https://www.teknophiles.com/2018/02/19/disk-pooling-in-linux-with-mergerfs/)
|
||||
- 2018-02-20 - [Fortes.com: Using Rclone and MergerFS together across drives](https://fortes.com/2018/rclone-and-mergerfs/)
|
||||
- 2019-02-10 - [Medium: Migrating from ZFS to MergerFS and SnapRAID at home](https://medium.com/@pascal.brokmeier/migrating-from-zfs-to-mergerfs-and-snapraid-at-home-89c45fd5db02)
|
||||
- 2019-04-24 - [MichaelXander.com: DIY NAS with OMV, SnapRAID, MergerFS, and Disk Encryption](https://michaelxander.com/diy-nas/)
|
||||
- 2019-07-16 - [Linuxserver.io: The Perfect Media Server - 2019 Edition](https://blog.linuxserver.io/2019/07/16/perfect-media-server-2019/)
|
||||
- 2019-09-10 - [Rclone VFS and MergerFS Setup](https://docs.usbx.me/books/rclone/page/rclone-vfs-and-mergerfs-setup)
|
||||
- 2019-12-20 - [NetworkShinobi.com: SnapRAID and MergerFS on OpenMediaVault](https://www.networkshinobi.com/snapraid-and-mergerfs-on-openmediavault/)
|
||||
- 2020-01-14 - [Brandon Rozek's Blog](https://brandonrozek.com/blog/mergerfs/)
|
||||
- 2020-02-14 - [SelfHostedHome.com: Combining Different Sized Drives with mergerfs and SnapRAID](https://selfhostedhome.com/combining-different-sized-drives-with-mergerfs-and-snapraid/)
|
||||
- 2020-05-01 - [FedoraMagazine.org: Using mergerfs to increase your virtual storage](https://fedoramagazine.org/using-mergerfs-to-increase-your-virtual-storage/)
|
||||
- 2020-08-20 - [Setting up Rclone, Mergerfs and Crontab for automated cloud storage](https://bytesized-hosting.com/pages/setting-up-rclone-mergerfs-and-crontab-for-automated-cloud-storage)
|
||||
- 2020-11-22 - [Introducing… MergerFS – My FREE UNRAID alternative](https://supertechfreaks.com/introducing-mergerfs-free-unraid-alternative/)
|
||||
- 2020-12-30 - [Perfect Media Server](https://perfectmediaserver.com) (a new site with docs fully fleshing out the 'Perfect Media Server' blog series)
|
||||
- 2021-07-24 - [Building the Ultimate Linux Home Server - Part 1: Intro, MergerFS, and SnapRAID](https://blog.karaolidis.com/ultimate-home-server-part-1/)
|
||||
- 2021-10-31 - [Better Home Storage: MergerFS + SnapRAID on OpenMediaVault](https://blog.sakuragawa.moe/better-home-storage-mergerfs-snapraid-on-openmediavault/)
|
||||
- 2021-11-28 - [Linux Magazine: Come Together - Merging file systems for a simple NAS with MergerFS](https://www.linux-magazine.com/Issues/2022/254/MergerFS)
|
||||
- 2022-06-04 - [MergerFS + SnapRaid Study](https://crashlaker.github.io/2022/06/04/mergerfs_+_snapraid_study.html)
|
||||
- 2022-12-31 - [Merge Storages in CasaOS: A secret beta feature you know now](https://blog.casaos.io/blog/13.html)
|
||||
- 2023-02-03 - [(MergerFS + SnapRAID) is the new RAID 5](https://thenomadcode.tech/mergerfs-snapraid-is-the-new-raid-5)
|
||||
- 2024-02-07 - [Designing & Deploying MANS - A Hybrid NAS Approach with SnapRAID, MergerFS, and OpenZFS](https://blog.muffn.io/posts/part-3-mini-100tb-nas)
|
||||
- 2024-03-11 - [Using MergerFS to combine multiple hard drives into one unified media storage](https://fullmetalbrackets.com/blog/two-drives-mergerfs/)
|
||||
- 2024-12-20 - [Pooling multiple drives on my Raspberry Pi with mergerfs](https://sebi.io/posts/2024-12-20-pooling-multiple-drives-with-mergerfs/)
|
||||
* 2016-02-02 - [Linuxserver.io: The Perfect Media Server 2016](https://blog.linuxserver.io/2016/02/02/the-perfect-media-server-2016/)
|
||||
* 2016-08-31 - [ZackReed.me: Mergerfs – another good option to pool your SnapRAID disks](https://zackreed.me/mergerfs-another-good-option-to-pool-your-snapraid-disks/)
|
||||
* 2016-11-06 - [Linuxserver.io: Revisiting the HP ProLiant Gen8 G1610T Microserver](https://blog.linuxserver.io/2016/11/06/revisiting-the-hp-proliant-gen8-g1610t-microserver/)
|
||||
* 2017-01-17 - [Setting up mergerfs on JBOD's (or a poor mans storage array)](http://corywestropp.com/develop/articles/setting-up-mergerfs/)
|
||||
* 2017-06-24 - [Linuxserver.io: The Perfect Media Server 2017](https://blog.linuxserver.io/2017/06/24/the-perfect-media-server-2017/)
|
||||
* 2018-02-19 - [Teknophiles: Disk Pooling in Linux with mergerFS](https://web.archive.org/web/20210324184857/https://www.teknophiles.com/2018/02/19/disk-pooling-in-linux-with-mergerfs/)
|
||||
* 2018-02-20 - [Fortes.com: Using Rclone and MergerFS together across drives](https://fortes.com/2018/rclone-and-mergerfs/)
|
||||
* 2019-02-10 - [Medium: Migrating from ZFS to MergerFS and SnapRAID at home](https://medium.com/@pascal.brokmeier/migrating-from-zfs-to-mergerfs-and-snapraid-at-home-89c45fd5db02)
|
||||
* 2019-04-24 - [MichaelXander.com: DIY NAS with OMV, SnapRAID, MergerFS, and Disk Encryption](https://michaelxander.com/diy-nas/)
|
||||
* 2019-07-16 - [Linuxserver.io: The Perfect Media Server - 2019 Edition](https://blog.linuxserver.io/2019/07/16/perfect-media-server-2019/)
|
||||
* 2019-09-10 - [Rclone VFS and MergerFS Setup](https://docs.usbx.me/books/rclone/page/rclone-vfs-and-mergerfs-setup)
|
||||
* 2019-12-20 - [NetworkShinobi.com: SnapRAID and MergerFS on OpenMediaVault](https://www.networkshinobi.com/snapraid-and-mergerfs-on-openmediavault/)
|
||||
* 2020-01-14 - [Brandon Rozek's Blog](https://brandonrozek.com/blog/mergerfs/)
|
||||
* 2020-02-14 - [SelfHostedHome.com: Combining Different Sized Drives with mergerfs and SnapRAID](https://selfhostedhome.com/combining-different-sized-drives-with-mergerfs-and-snapraid/)
|
||||
* 2020-05-01 - [FedoraMagazine.org: Using mergerfs to increase your virtual storage](https://fedoramagazine.org/using-mergerfs-to-increase-your-virtual-storage/)
|
||||
* 2020-08-20 - [Setting up Rclone, Mergerfs and Crontab for automated cloud storage](https://bytesized-hosting.com/pages/setting-up-rclone-mergerfs-and-crontab-for-automated-cloud-storage)
|
||||
* 2020-11-22 - [Introducing… MergerFS – My FREE UNRAID alternative](https://supertechfreaks.com/introducing-mergerfs-free-unraid-alternative/)
|
||||
* 2020-12-30 - [Perfect Media Server](https://perfectmediaserver.com) (a new site with docs fully fleshing out the 'Perfect Media Server' blog series)
|
||||
* 2021-07-24 - [Building the Ultimate Linux Home Server - Part 1: Intro, MergerFS, and SnapRAID](https://blog.karaolidis.com/ultimate-home-server-part-1/)
|
||||
* 2021-10-31 - [Better Home Storage: MergerFS + SnapRAID on OpenMediaVault](https://blog.sakuragawa.moe/better-home-storage-mergerfs-snapraid-on-openmediavault/)
|
||||
* 2021-11-28 - [Linux Magazine: Come Together - Merging file systems for a simple NAS with MergerFS](https://www.linux-magazine.com/Issues/2022/254/MergerFS)
|
||||
* 2022-06-04 - [MergerFS + SnapRaid Study](https://crashlaker.github.io/2022/06/04/mergerfs_+_snapraid_study.html)
|
||||
* 2022-12-31 - [Merge Storages in CasaOS: A secret beta feature you know now](https://blog.casaos.io/blog/13.html)
|
||||
* 2023-02-03 - [(MergerFS + SnapRAID) is the new RAID 5](https://thenomadcode.tech/mergerfs-snapraid-is-the-new-raid-5)
|
||||
* 2024-02-07 - [Designing & Deploying MANS - A Hybrid NAS Approach with SnapRAID, MergerFS, and OpenZFS](https://blog.muffn.io/posts/part-3-mini-100tb-nas)
|
||||
* 2024-03-11 - [Using MergerFS to combine multiple hard drives into one unified media storage](https://fullmetalbrackets.com/blog/two-drives-mergerfs/)
|
||||
* 2024-12-20 - [Pooling multiple drives on my Raspberry Pi with mergerfs](https://sebi.io/posts/2024-12-20-pooling-multiple-drives-with-mergerfs/)
|
||||
|
||||
|
||||
## Videos
|
||||
|
||||
- 2017-06-23 - [Alex Kretzschmar: Part 1 - Perfect Media Server 2017 - Introduction](https://www.youtube.com/watch?v=L5MH8q3lmmk)
|
||||
- 2017-06-24 - [Alex Kretzschmar: Part 2 - Perfect Media server 2017 - Installing Debian 9 Stretch](https://www.youtube.com/watch?v=YpVVYRN_L_A)
|
||||
- 2017-06-24 - [Alex Kretzschmar: Part 3 - Perfect Media Server 2017 - Install MergerFS and setting up your drives](https://www.youtube.com/watch?v=tbCMfm-jJ5Y)
|
||||
- 2017-06-24 - [Alex Kretzschmar: Part 4 - Perfect Media Server 2017 - Installing Docker](https://www.youtube.com/watch?v=WYI32kx4hPE)
|
||||
- 2017-06-24 - [Alex Kretzschmar: Part 5 - Perfect Media Server 2017 - Installing and Automating SnapRAID](https://www.youtube.com/watch?v=Ir5ZsUIbHXA)
|
||||
- 2017-06-24 - [Alex Kretzschmar: Part 6 - Perfect Media Server 2017 -Turning your server into a NAS with Samba and NFS](https://www.youtube.com/watch?v=1hVdWq758ZQ)
|
||||
- 2017-06-24 - [Alex Kretzschmar: Part 7 - Perfect Media Server 2017 - Managing your apps with docker-compose](https://www.youtube.com/watch?v=aI2rdw7_AmE)
|
||||
- 2017-06-24 - [Alex Kretzschmar: Part 8 - Perfect Media Server 2017 - Manging your server using a web UI (Cockpit and Portainer)](https://www.youtube.com/watch?v=aLyTWdzDiCg)
|
||||
- 2018-04-24 - [ElectronicsWizardy: How to setup a Linux Fileserver with Snapraid and Mergerfs Re-Export](https://www.youtube.com/watch?v=D2Klx-X7pFo)
|
||||
- 2019-03-01 - [Snapraid and Unionfs: Advanced Array Options on Openmediavault (Better than ZFS and Unraid)](https://www.youtube.com/watch?v=FYkdPyCt5FU)
|
||||
- 2019-03-22 - [Gamexplicit: The Perfect Plex Media Server 2019! Part 1 (Hardware)](https://www.youtube.com/watch?v=rJIRPhM2WcE)
|
||||
- 2019-05-13 - [Gamexplicit: The Perfect Plex Media Server 2019! Part 2 (Ubuntu Server 18.04.2 LTS)](https://www.youtube.com/watch?v=aLyTWdzDiCg)
|
||||
- 2019-05-20 - [Gamexplicit: The Perfect Plex Media Server 2019! Part 3 (SnapRaid, Samba, Plex)](https://www.youtube.com/watch?v=uW5y43XC-BI)
|
||||
- 2020-08-23 - [Installing OpenMediaVault and SnapRAID and UnionFS (mergerfs)](https://www.youtube.com/watch?v=nDvzXM8UjAI)
|
||||
- 2021-06-07 - [I ditched TrueNAS for MergerFS - Chia Plot Storage](https://www.youtube.com/watch?v=tpqFywkbZa4)
|
||||
- 2021-08-03 - [Install and configure mergerfs to merge more than one folder in the same place](https://www.youtube.com/watch?v=69zcqEy1674)
|
||||
- 2021-08-10 - [Instalando e configurando mergerfs para unir mais de uma pasta no mesmo lugar](https://www.youtube.com/watch?v=-RLxbBNBWhU)
|
||||
- 2021-08-13 - [Let's Convert an Old Laptop to a NAS - What you should do](https://www.youtube.com/watch?v=F1v-TSbOymI)
|
||||
- 2021-08-17 - [How to Combine Multiple Disks as One by using MergerFS | Ubuntu 20.04 LTS](https://www.youtube.com/watch?v=9e46pz5Seo4)
|
||||
- 2021-08-20 - [Vamos converter um Laptop antigo para um NAS – O que você deve fazer](https://www.youtube.com/watch?v=q8EK9vWCRTc)
|
||||
- 2021-10-29 - [Unlimited space for your Plex using Rclone to connect to your Cloud](https://www.youtube.com/watch?v=ghGconyrF3M)
|
||||
- 2022-12-01 - [Make Your Home Server Go FAST! SSD Caching, 10Gbit Networking, etc.](https://www.youtube.com/watch?v=eRfqC_q3lkM&t=784s)
|
||||
- 2022-12-07 - [Best RAID for mixed drive sizes. Unraid vs BTRFS vs Snapraid+Mergerfs vs Storage spaces.](https://www.youtube.com/watch?v=NQJkTiLXfgs)
|
||||
- 2023-02-21 - [MergerFS + SnapRAID : Forget about RAID 5 in your Home Server !](https://www.youtube.com/watch?v=tX5MA-c6Qq4)
|
||||
- 2023-06-26 - [How to install and setup MergerFS](https://www.youtube.com/watch?v=n7piuhTXeG4)
|
||||
- 2023-07-31 - [How to recover a dead drive using Snapraid](https://www.youtube.com/watch?v=fmuiRLPcuJE)
|
||||
- 2024-01-05 - [OpenMediaVault MergerFS Tutorial (Portuguese)](https://www.youtube.com/watch?v=V6Yw86dRUPQ)
|
||||
- 2024-02-19 - [Setup and Install MergerFS and SnapRAID (Part 1)](https://noted.lol/mergerfs-and-snapraid-setup-1/)
|
||||
- 2024-02-22 - [Setup and Install MergerFS and SnapRAID (Part 2)](https://noted.lol/mergerfs-and-snapraid-setup-part-2/)
|
||||
- 2024-11-15 - [Meu servidor NAS - Parte 18: Recuperando um HD, recuperando o MergerFS e os próximos passos do NAS!](https://www.youtube.com/watch?v=5fy98kPzE3s)
|
||||
|
||||
* 2017-06-23 - [Alex Kretzschmar: Part 1 - Perfect Media Server 2017 - Introduction](https://www.youtube.com/watch?v=L5MH8q3lmmk)
|
||||
* 2017-06-24 - [Alex Kretzschmar: Part 2 - Perfect Media server 2017 - Installing Debian 9 Stretch](https://www.youtube.com/watch?v=YpVVYRN_L_A)
|
||||
* 2017-06-24 - [Alex Kretzschmar: Part 3 - Perfect Media Server 2017 - Install MergerFS and setting up your drives](https://www.youtube.com/watch?v=tbCMfm-jJ5Y)
|
||||
* 2017-06-24 - [Alex Kretzschmar: Part 4 - Perfect Media Server 2017 - Installing Docker](https://www.youtube.com/watch?v=WYI32kx4hPE)
|
||||
* 2017-06-24 - [Alex Kretzschmar: Part 5 - Perfect Media Server 2017 - Installing and Automating SnapRAID](https://www.youtube.com/watch?v=Ir5ZsUIbHXA)
|
||||
* 2017-06-24 - [Alex Kretzschmar: Part 6 - Perfect Media Server 2017 -Turning your server into a NAS with Samba and NFS](https://www.youtube.com/watch?v=1hVdWq758ZQ)
|
||||
* 2017-06-24 - [Alex Kretzschmar: Part 7 - Perfect Media Server 2017 - Managing your apps with docker-compose](https://www.youtube.com/watch?v=aI2rdw7_AmE)
|
||||
* 2017-06-24 - [Alex Kretzschmar: Part 8 - Perfect Media Server 2017 - Manging your server using a web UI (Cockpit and Portainer)](https://www.youtube.com/watch?v=aLyTWdzDiCg)
|
||||
* 2018-04-24 - [ElectronicsWizardy: How to setup a Linux Fileserver with Snapraid and Mergerfs Re-Export](https://www.youtube.com/watch?v=D2Klx-X7pFo)
|
||||
* 2019-03-01 - [Snapraid and Unionfs: Advanced Array Options on Openmediavault (Better than ZFS and Unraid)](https://www.youtube.com/watch?v=FYkdPyCt5FU)
|
||||
* 2019-03-22 - [Gamexplicit: The Perfect Plex Media Server 2019! Part 1 (Hardware)](https://www.youtube.com/watch?v=rJIRPhM2WcE)
|
||||
* 2019-05-13 - [Gamexplicit: The Perfect Plex Media Server 2019! Part 2 (Ubuntu Server 18.04.2 LTS)](https://www.youtube.com/watch?v=aLyTWdzDiCg)
|
||||
* 2019-05-20 - [Gamexplicit: The Perfect Plex Media Server 2019! Part 3 (SnapRaid, Samba, Plex)](https://www.youtube.com/watch?v=uW5y43XC-BI)
|
||||
* 2020-08-23 - [Installing OpenMediaVault and SnapRAID and UnionFS (mergerfs)](https://www.youtube.com/watch?v=nDvzXM8UjAI)
|
||||
* 2021-06-07 - [I ditched TrueNAS for MergerFS - Chia Plot Storage](https://www.youtube.com/watch?v=tpqFywkbZa4)
|
||||
* 2021-08-03 - [Install and configure mergerfs to merge more than one folder in the same place](https://www.youtube.com/watch?v=69zcqEy1674)
|
||||
* 2021-08-10 - [Instalando e configurando mergerfs para unir mais de uma pasta no mesmo lugar](https://www.youtube.com/watch?v=-RLxbBNBWhU)
|
||||
* 2021-08-13 - [Let's Convert an Old Laptop to a NAS - What you should do](https://www.youtube.com/watch?v=F1v-TSbOymI)
|
||||
* 2021-08-17 - [How to Combine Multiple Disks as One by using MergerFS | Ubuntu 20.04 LTS](https://www.youtube.com/watch?v=9e46pz5Seo4)
|
||||
* 2021-08-20 - [Vamos converter um Laptop antigo para um NAS – O que você deve fazer](https://www.youtube.com/watch?v=q8EK9vWCRTc)
|
||||
* 2021-10-29 - [Unlimited space for your Plex using Rclone to connect to your Cloud](https://www.youtube.com/watch?v=ghGconyrF3M)
|
||||
* 2022-12-01 - [Make Your Home Server Go FAST! SSD Caching, 10Gbit Networking, etc.](https://www.youtube.com/watch?v=eRfqC_q3lkM&t=784s)
|
||||
* 2022-12-07 - [Best RAID for mixed drive sizes. Unraid vs BTRFS vs Snapraid+Mergerfs vs Storage spaces.](https://www.youtube.com/watch?v=NQJkTiLXfgs)
|
||||
* 2023-02-21 - [MergerFS + SnapRAID : Forget about RAID 5 in your Home Server !](https://www.youtube.com/watch?v=tX5MA-c6Qq4)
|
||||
* 2023-06-26 - [How to install and setup MergerFS](https://www.youtube.com/watch?v=n7piuhTXeG4)
|
||||
* 2023-07-31 - [How to recover a dead drive using Snapraid](https://www.youtube.com/watch?v=fmuiRLPcuJE)
|
||||
* 2024-01-05 - [OpenMediaVault MergerFS Tutorial (Portuguese)](https://www.youtube.com/watch?v=V6Yw86dRUPQ)
|
||||
* 2024-02-19 - [Setup and Install MergerFS and SnapRAID (Part 1)](https://noted.lol/mergerfs-and-snapraid-setup-1/)
|
||||
* 2024-02-22 - [Setup and Install MergerFS and SnapRAID (Part 2)](https://noted.lol/mergerfs-and-snapraid-setup-part-2/)
|
||||
* 2024-11-15 - [Meu servidor NAS - Parte 18: Recuperando um HD, recuperando o MergerFS e os próximos passos do NAS!](https://www.youtube.com/watch?v=5fy98kPzE3s)
|
||||
* 2025-04-23 - [How to build the Perfect Media Server | Part 1 - The Tech Stack | mergerfs, SnapRAID, and docker.](https://www.youtube.com/watch?v=Yt67zz9p0FU)
|
||||
|
||||
## Podcasts
|
||||
|
||||
- 2019-11-04 - [Jupiter Extras: A Chat with mergerfs Developer Antonio Musumeci | Jupiter Extras 28](https://www.youtube.com/watch?v=VmJUAyyhSPk)
|
||||
- 2019-11-07 - [Jupiter Broadcasting: ZFS Isn’t the Only Option | Self-Hosted 5](https://www.youtube.com/watch?v=JEW7UuKhMJ8)
|
||||
- 2023-10-08 - [Self Hosted Episode 105 - Sleeper Storage Technology](https://selfhosted.show/105)
|
||||
* 2019-11-04 - [Jupiter Extras: A Chat with mergerfs Developer Antonio Musumeci | Jupiter Extras 28](https://www.youtube.com/watch?v=VmJUAyyhSPk)
|
||||
* 2019-11-07 - [Jupiter Broadcasting: ZFS Isn’t the Only Option | Self-Hosted 5](https://www.youtube.com/watch?v=JEW7UuKhMJ8)
|
||||
* 2023-10-08 - [Self Hosted Episode 105 - Sleeper Storage Technology](https://selfhosted.show/105)
|
||||
|
||||
|
||||
## Social Media
|
||||
|
||||
- [Reddit](https://www.reddit.com/search/?q=mergerfs&sort=new)
|
||||
- [X](https://x.com/search?q=mergerfs&src=spelling_expansion_revert_click&f=live)
|
||||
- [YouTube](https://www.youtube.com/results?search_query=mergerfs&sp=CAI%253D)
|
||||
- [ServeTheHome Forum](https://forums.servethehome.com/index.php?search/3105813/&q=mergerfs&o=date)
|
||||
* [Reddit](https://www.reddit.com/search/?q=mergerfs&sort=new)
|
||||
* [X](https://x.com/search?q=mergerfs&src=spelling_expansion_revert_click&f=live)
|
||||
* [YouTube](https://www.youtube.com/results?search_query=mergerfs&sp=CAI%253D)
|
||||
* [ServeTheHome Forum](https://forums.servethehome.com/index.php?search/3105813/&q=mergerfs&o=date)
|
||||
|
@ -59,6 +59,10 @@ IO.)
|
||||
|
||||
## Usage
|
||||
|
||||
For some suggestions on branch setup see [the page on
|
||||
branches](config/branches.md#branch-setup).
|
||||
|
||||
|
||||
### Command Line
|
||||
|
||||
```
|
||||
|
@ -4,14 +4,15 @@
|
||||
|
||||
Some storage technologies support what is called "tiered" caching. The
|
||||
placing of smaller, faster storage as a transparent cache to larger,
|
||||
slower storage. NVMe, SSD, Optane in front of traditional HDDs for
|
||||
slower storage. NVMe, SSD, or Optane in front of traditional HDDs for
|
||||
instance.
|
||||
|
||||
mergerfs does not natively support any sort of tiered caching. Most
|
||||
users have no use for such a feature and its inclusion would
|
||||
complicate the code as it exists today. However, there are a few
|
||||
situations where a cache filesystem could help with a typical mergerfs
|
||||
setup.
|
||||
mergerfs does not natively support any sort of tiered caching
|
||||
currently. The truth is for many users a cache would have little or no
|
||||
advantage over reading and writing directly. They would be
|
||||
bottlenecked by their network, internet connection, or limited size of
|
||||
the cache. However, there are a few situations where a tiered cache
|
||||
setup could help.
|
||||
|
||||
1. Fast network, slow filesystems, many readers: You've a 10+Gbps
|
||||
network with many readers and your regular filesystems can't keep
|
||||
@ -34,7 +35,7 @@ dm-cache but there is another solution which requires only mergerfs, a
|
||||
script to move files around, and a cron job to run said script.
|
||||
|
||||
* Create two mergerfs pools. One which includes just the **slow**
|
||||
branches and one which has both the **fast** branches
|
||||
branches and one which has **both** the **fast** branches
|
||||
(SSD,NVME,etc.) and **slow** branches. The **base** pool and the
|
||||
**cache** pool.
|
||||
* The **cache** pool should have the cache branches listed first in
|
||||
@ -43,7 +44,7 @@ script to move files around, and a cron job to run said script.
|
||||
probably be `ff`, `lus`, or `lfs`. The latter two under the
|
||||
assumption that the cache filesystem(s) are far smaller than the
|
||||
backing filesystems.
|
||||
* You can also set the **slow** filesystems mode to `NC` which would
|
||||
* You can also set the **slow** branches' mode to `NC` which would
|
||||
give you the ability to use other `create` policies though that'd
|
||||
mean if the cache filesystems fill you'd get "out of space"
|
||||
errors. This however may be good as it would indicate the script
|
||||
|
@ -5,6 +5,9 @@ repo_name: mergerfs
|
||||
repo_url: https://github.com/trapexit/mergerfs
|
||||
edit_uri: edit/master/mkdocs/docs/
|
||||
docs_dir: docs
|
||||
plugins:
|
||||
- search
|
||||
- tags
|
||||
theme:
|
||||
name: material
|
||||
logo: logo.png
|
||||
@ -62,6 +65,7 @@ nav:
|
||||
- config/options.md
|
||||
- config/deprecated_options.md
|
||||
- config/branches.md
|
||||
- config/branches_mount_timeout.md
|
||||
- config/functions_categories_policies.md
|
||||
- config/minfreespace.md
|
||||
- config/func_readdir.md
|
||||
|
@ -56,6 +56,7 @@ namespace l
|
||||
{
|
||||
IFERT("async_read");
|
||||
IFERT("branches-mount-timeout");
|
||||
IFERT("branches-mount-timeout-fail");
|
||||
IFERT("cache.symlinks");
|
||||
IFERT("cache.writeback");
|
||||
IFERT("direct-io-allow-mmap");
|
||||
@ -85,6 +86,7 @@ Config::Config()
|
||||
minfreespace(MINFREESPACE_DEFAULT),
|
||||
branches(minfreespace),
|
||||
branches_mount_timeout(0),
|
||||
branches_mount_timeout_fail(false),
|
||||
cache_attr(1),
|
||||
cache_entry(1),
|
||||
cache_files(CacheFiles::ENUM::OFF),
|
||||
@ -139,6 +141,7 @@ Config::Config()
|
||||
_map["auto_cache"] = &auto_cache;
|
||||
_map["branches"] = &branches;
|
||||
_map["branches-mount-timeout"] = &branches_mount_timeout;
|
||||
_map["branches-mount-timeout-fail"] = &branches_mount_timeout_fail;
|
||||
_map["cache.attr"] = &cache_attr;
|
||||
_map["cache.entry"] = &cache_entry;
|
||||
_map["cache.files"] = &cache_files;
|
||||
|
@ -41,6 +41,8 @@
|
||||
#include "rwlock.hpp"
|
||||
#include "tofrom_wrapper.hpp"
|
||||
|
||||
#include "ghc/filesystem.hpp"
|
||||
|
||||
#include "fuse.h"
|
||||
|
||||
#include <cstdint>
|
||||
@ -55,6 +57,7 @@ typedef ToFromWrapper<bool> ConfigBOOL;
|
||||
typedef ToFromWrapper<uint64_t> ConfigUINT64;
|
||||
typedef ToFromWrapper<int> ConfigINT;
|
||||
typedef ToFromWrapper<std::string> ConfigSTR;
|
||||
typedef ToFromWrapper<ghc::filesystem::path> ConfigPath;
|
||||
typedef std::map<std::string,ToFromString*> Str2TFStrMap;
|
||||
|
||||
extern const std::string CONTROLFILE;
|
||||
@ -108,6 +111,7 @@ public:
|
||||
ConfigUINT64 minfreespace;
|
||||
Branches branches;
|
||||
ConfigUINT64 branches_mount_timeout;
|
||||
ConfigBOOL branches_mount_timeout_fail;
|
||||
ConfigUINT64 cache_attr;
|
||||
ConfigUINT64 cache_entry;
|
||||
CacheFiles cache_files;
|
||||
@ -133,7 +137,7 @@ public:
|
||||
ConfigBOOL link_cow;
|
||||
LinkEXDEV link_exdev;
|
||||
LogMetrics log_metrics;
|
||||
ConfigSTR mountpoint;
|
||||
ConfigPath mountpoint;
|
||||
MoveOnENOSPC moveonenospc;
|
||||
NFSOpenHack nfsopenhack;
|
||||
ConfigBOOL nullrw;
|
||||
|
@ -16,12 +16,11 @@
|
||||
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
|
||||
*/
|
||||
|
||||
#include "from_string.hpp"
|
||||
|
||||
#include "ef.hpp"
|
||||
#include "errno.hpp"
|
||||
|
||||
#include <cstdint>
|
||||
#include <string>
|
||||
|
||||
#include <stdlib.h>
|
||||
|
||||
|
||||
@ -128,4 +127,13 @@ namespace str
|
||||
{
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
int
|
||||
from(const std::string &value_,
|
||||
fs::Path *path_)
|
||||
{
|
||||
*path_ = value_;
|
||||
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
@ -18,6 +18,8 @@
|
||||
|
||||
#pragma once
|
||||
|
||||
#include "fs_path.hpp"
|
||||
|
||||
#include <cstdint>
|
||||
#include <string>
|
||||
|
||||
@ -29,4 +31,5 @@ namespace str
|
||||
int from(const std::string &, uint64_t *);
|
||||
int from(const std::string &, std::string *);
|
||||
int from(const std::string &, const std::string *);
|
||||
int from(const std::string &, fs::Path *);
|
||||
}
|
||||
|
@ -18,7 +18,7 @@
|
||||
|
||||
#pragma once
|
||||
|
||||
#include "ghc/filesystem.hpp"
|
||||
#include "fs_path.hpp"
|
||||
|
||||
#include <string>
|
||||
|
||||
@ -49,7 +49,7 @@ namespace fs
|
||||
static
|
||||
inline
|
||||
int
|
||||
mkdir(const ghc::filesystem::path &path_,
|
||||
mkdir(const fs::Path &path_,
|
||||
const mode_t mode_)
|
||||
{
|
||||
return fs::mkdir(path_.c_str(),mode_);
|
||||
|
@ -57,8 +57,8 @@ namespace l
|
||||
namespace fs
|
||||
{
|
||||
std::tuple<int,std::string>
|
||||
mktemp_in_dir(std::string const dirpath_,
|
||||
int const flags_)
|
||||
mktemp_in_dir(const std::string dirpath_,
|
||||
const int flags_)
|
||||
{
|
||||
int fd;
|
||||
int count;
|
||||
@ -85,10 +85,10 @@ namespace fs
|
||||
}
|
||||
|
||||
std::tuple<int,std::string>
|
||||
mktemp(std::string const filepath_,
|
||||
int const flags_)
|
||||
mktemp(const std::string filepath_,
|
||||
const int flags_)
|
||||
{
|
||||
ghc::filesystem::path filepath{filepath_};
|
||||
fs::Path filepath{filepath_};
|
||||
|
||||
return fs::mktemp_in_dir(filepath.parent_path(),flags_);
|
||||
}
|
||||
|
37
src/fs_mounts.cpp
Normal file
37
src/fs_mounts.cpp
Normal file
@ -0,0 +1,37 @@
|
||||
#include "fs_mounts.hpp"
|
||||
|
||||
#include <cstdio>
|
||||
|
||||
#include <mntent.h>
|
||||
|
||||
#ifdef __linux__
|
||||
void
|
||||
fs::mounts(fs::MountVec &mounts_)
|
||||
{
|
||||
FILE *f;
|
||||
|
||||
f = setmntent("/proc/mounts","r");
|
||||
if(f == NULL)
|
||||
return;
|
||||
|
||||
struct mntent *entry;
|
||||
while((entry = getmntent(f)) != NULL)
|
||||
{
|
||||
fs::Mount m;
|
||||
|
||||
m.dir = entry->mnt_dir;
|
||||
m.fsname = entry->mnt_fsname;
|
||||
m.type = entry->mnt_type;
|
||||
m.opts = entry->mnt_opts;
|
||||
|
||||
mounts_.emplace_back(std::move(m));
|
||||
}
|
||||
|
||||
endmntent(f);
|
||||
}
|
||||
#else
|
||||
void
|
||||
fs::mounts(fs::MountVec &mounts_)
|
||||
{
|
||||
}
|
||||
#endif
|
21
src/fs_mounts.hpp
Normal file
21
src/fs_mounts.hpp
Normal file
@ -0,0 +1,21 @@
|
||||
#pragma once
|
||||
|
||||
#include "fs_path.hpp"
|
||||
|
||||
#include <vector>
|
||||
|
||||
|
||||
namespace fs
|
||||
{
|
||||
struct Mount
|
||||
{
|
||||
fs::Path dir;
|
||||
std::string fsname;
|
||||
std::string type;
|
||||
std::string opts;
|
||||
};
|
||||
|
||||
typedef std::vector<fs::Mount> MountVec;
|
||||
|
||||
void mounts(fs::MountVec &mounts);
|
||||
}
|
@ -19,6 +19,10 @@
|
||||
#include "fs_wait_for_mount.hpp"
|
||||
#include "syslog.hpp"
|
||||
|
||||
#include "fs_exists.hpp"
|
||||
#include "fs_lstat.hpp"
|
||||
#include "fs_lgetxattr.hpp"
|
||||
|
||||
#include <functional>
|
||||
#include <thread>
|
||||
#include <unordered_set>
|
||||
@ -44,6 +48,40 @@ namespace std
|
||||
};
|
||||
}
|
||||
|
||||
static
|
||||
bool
|
||||
_branch_is_mounted(const struct stat &src_st_,
|
||||
const fs::Path &branch_path_)
|
||||
{
|
||||
int rv;
|
||||
struct stat st;
|
||||
fs::Path filepath;
|
||||
|
||||
rv = fs::lgetxattr(branch_path_,"user.mergerfs.branch",NULL,0);
|
||||
if(rv != -1)
|
||||
return true;
|
||||
|
||||
filepath = branch_path_ / ".mergerfs.branch";
|
||||
rv = fs::exists(filepath);
|
||||
if(rv)
|
||||
return true;
|
||||
|
||||
rv = fs::lgetxattr(branch_path_,"user.mergerfs.branch_mounts_here",NULL,0);
|
||||
if(rv != -1)
|
||||
return false;
|
||||
|
||||
filepath = branch_path_ / ".mergerfs.branch_mounts_here";
|
||||
rv = fs::exists(filepath);
|
||||
if(rv)
|
||||
return false;
|
||||
|
||||
rv = fs::lstat(branch_path_,&st);
|
||||
if(rv == 0)
|
||||
return (st.st_dev != src_st_.st_dev);
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
static
|
||||
void
|
||||
_check_mounted(const struct stat &src_st_,
|
||||
@ -56,30 +94,23 @@ _check_mounted(const struct stat &src_st_,
|
||||
|
||||
for(auto const &tgt_path : tgt_paths_)
|
||||
{
|
||||
int rv;
|
||||
struct stat tgt_st;
|
||||
bool mounted;
|
||||
|
||||
rv = fs::stat(tgt_path,&tgt_st);
|
||||
if(rv == 0)
|
||||
{
|
||||
if(tgt_st.st_dev != src_st_.st_dev)
|
||||
mounted = ::_branch_is_mounted(src_st_,tgt_path);
|
||||
if(mounted)
|
||||
successes.push_back(tgt_path);
|
||||
else
|
||||
failures.push_back(tgt_path);
|
||||
}
|
||||
else
|
||||
{
|
||||
failures.push_back(tgt_path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static
|
||||
void
|
||||
int
|
||||
_wait_for_mount(const struct stat &src_st_,
|
||||
const fs::PathVector &tgt_paths_,
|
||||
const std::chrono::milliseconds &timeout_)
|
||||
{
|
||||
bool first_loop;
|
||||
fs::PathVector successes;
|
||||
fs::PathVector failures;
|
||||
std::unordered_set<fs::Path> tgt_paths;
|
||||
@ -90,6 +121,7 @@ _wait_for_mount(const struct stat &src_st_,
|
||||
now = std::chrono::steady_clock::now();
|
||||
deadline = now + timeout_;
|
||||
|
||||
first_loop = true;
|
||||
while(true)
|
||||
{
|
||||
if(tgt_paths.empty())
|
||||
@ -100,10 +132,17 @@ _wait_for_mount(const struct stat &src_st_,
|
||||
successes.clear();
|
||||
failures.clear();
|
||||
::_check_mounted(src_st_,tgt_paths,&successes,&failures);
|
||||
for(auto const &path : successes)
|
||||
for(const auto &path : successes)
|
||||
{
|
||||
tgt_paths.erase(path);
|
||||
syslog_info("%s is mounted",path.string().c_str());
|
||||
syslog_info("%s is mounted",path.c_str());
|
||||
}
|
||||
|
||||
if(first_loop)
|
||||
{
|
||||
for(const auto &path : failures)
|
||||
syslog_notice("%s is not mounted, waiting",path.c_str());
|
||||
first_loop = false;
|
||||
}
|
||||
|
||||
std::this_thread::sleep_for(SLEEP_DURATION);
|
||||
@ -111,26 +150,24 @@ _wait_for_mount(const struct stat &src_st_,
|
||||
}
|
||||
|
||||
for(auto const &path : failures)
|
||||
syslog_notice("%s not mounted within timeout",path.string().c_str());
|
||||
if(!failures.empty())
|
||||
syslog_warning("Continuing to mount mergerfs despite %u branches not "
|
||||
"being different from the mountpoint filesystem",
|
||||
failures.size());
|
||||
syslog_notice("%s not mounted within timeout",path.c_str());
|
||||
|
||||
return failures.size();
|
||||
}
|
||||
|
||||
void
|
||||
int
|
||||
fs::wait_for_mount(const fs::Path &src_path_,
|
||||
const fs::PathVector &tgt_paths_,
|
||||
const std::chrono::milliseconds &timeout_)
|
||||
{
|
||||
int rv;
|
||||
struct stat src_st;
|
||||
struct stat src_st = {0};
|
||||
|
||||
rv = fs::stat(src_path_,&src_st);
|
||||
if(rv == -1)
|
||||
return syslog_error("Error stat'ing mount path: %s (%s)",
|
||||
syslog_error("Error stat'ing mount path: %s (%s)",
|
||||
src_path_.c_str(),
|
||||
strerror(errno));
|
||||
|
||||
::_wait_for_mount(src_st,tgt_paths_,timeout_);
|
||||
return ::_wait_for_mount(src_st,tgt_paths_,timeout_);
|
||||
}
|
||||
|
@ -1,6 +1,5 @@
|
||||
/*
|
||||
ISC License
|
||||
|
||||
Copyright (c) 2023, Antonio SJ Musumeci <trapexit@spawn.link>
|
||||
|
||||
Permission to use, copy, modify, and/or distribute this software for any
|
||||
@ -29,7 +28,7 @@
|
||||
|
||||
namespace fs
|
||||
{
|
||||
void
|
||||
int
|
||||
wait_for_mount(const fs::Path &srcpath,
|
||||
const fs::PathVector &tgtpaths,
|
||||
const std::chrono::milliseconds &timeout);
|
||||
|
@ -19,12 +19,13 @@
|
||||
#include "fs_readahead.hpp"
|
||||
#include "syslog.hpp"
|
||||
|
||||
#include "fs_path.hpp"
|
||||
#include "fs_exists.hpp"
|
||||
|
||||
#include "fmt/core.h"
|
||||
|
||||
#include "fuse.h"
|
||||
|
||||
#include "ghc/filesystem.hpp"
|
||||
|
||||
#include <thread>
|
||||
#include <algorithm>
|
||||
|
||||
@ -82,7 +83,7 @@ namespace l
|
||||
std::fstream f;
|
||||
uint64_t max_pages_limit;
|
||||
|
||||
if(ghc::filesystem::exists(MAX_PAGES_LIMIT_FILEPATH))
|
||||
if(fs::exists(MAX_PAGES_LIMIT_FILEPATH))
|
||||
{
|
||||
if(cfg_->fuse_msg_size > MAX_FUSE_MSG_SIZE)
|
||||
syslog_info("fuse_msg_size > %u: setting it to %u",
|
||||
@ -141,7 +142,7 @@ namespace l
|
||||
|
||||
static
|
||||
void
|
||||
readahead(const std::string path_,
|
||||
readahead(const fs::Path path_,
|
||||
const int readahead_)
|
||||
{
|
||||
int rv;
|
||||
|
@ -32,7 +32,6 @@
|
||||
|
||||
using std::string;
|
||||
using std::vector;
|
||||
namespace gfs = ghc::filesystem;
|
||||
|
||||
namespace error
|
||||
{
|
||||
@ -230,8 +229,8 @@ namespace l
|
||||
fuse_timeouts_t *timeouts_)
|
||||
{
|
||||
int rv;
|
||||
gfs::path target(oldpath_);
|
||||
gfs::path linkpath(newpath_);
|
||||
fs::Path target(oldpath_);
|
||||
fs::Path linkpath(newpath_);
|
||||
|
||||
target = target.lexically_relative(linkpath.parent_path());
|
||||
|
||||
@ -278,7 +277,7 @@ namespace l
|
||||
|
||||
static
|
||||
int
|
||||
link_exdev_abs_pool_symlink(const std::string &mount_,
|
||||
link_exdev_abs_pool_symlink(const fs::Path mount_,
|
||||
const char *oldpath_,
|
||||
const char *newpath_,
|
||||
struct stat *st_,
|
||||
|
@ -315,7 +315,7 @@ namespace l
|
||||
int
|
||||
rename_exdev_abs_symlink(const Policy::Action &actionPolicy_,
|
||||
const Branches::CPtr &branches_,
|
||||
const std::string &mount_,
|
||||
const gfs::path &mount_,
|
||||
const gfs::path &oldfusepath_,
|
||||
const gfs::path &newfusepath_)
|
||||
{
|
||||
|
@ -165,26 +165,51 @@ namespace l
|
||||
}
|
||||
|
||||
static
|
||||
void
|
||||
bool
|
||||
wait_for_mount(const Config::Read &cfg_)
|
||||
{
|
||||
int failures;
|
||||
fs::PathVector paths;
|
||||
std::chrono::milliseconds timeout;
|
||||
|
||||
paths = cfg_->branches->to_paths();
|
||||
|
||||
syslog_info("Waiting %u seconds for branches to mount",
|
||||
(uint64_t)cfg_->branches_mount_timeout);
|
||||
syslog_info("Waiting %u seconds for %zu branches to mount",
|
||||
(uint64_t)cfg_->branches_mount_timeout,
|
||||
paths.size());
|
||||
|
||||
timeout = std::chrono::milliseconds(cfg_->branches_mount_timeout * 1000);
|
||||
fs::wait_for_mount((std::string)cfg_->mountpoint,
|
||||
failures = fs::wait_for_mount(cfg_->mountpoint,
|
||||
paths,
|
||||
timeout);
|
||||
if(failures)
|
||||
{
|
||||
if(cfg_->branches_mount_timeout_fail)
|
||||
{
|
||||
syslog_error("%d of %zu branches were not mounted"
|
||||
" within the timeout of %zus. Exiting",
|
||||
failures,
|
||||
paths.size(),
|
||||
(uint64_t)cfg_->branches_mount_timeout);
|
||||
return true;
|
||||
}
|
||||
|
||||
syslog_warning("Continuing to mount mergerfs despite %d branches not "
|
||||
"being different from the mountpoint filesystem",
|
||||
failures);
|
||||
}
|
||||
else
|
||||
{
|
||||
syslog_info("All %zd branches are mounted",
|
||||
paths.size());
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
static
|
||||
void
|
||||
lazy_umount(const std::string target_)
|
||||
lazy_umount(const fs::Path target_)
|
||||
{
|
||||
int rv;
|
||||
|
||||
@ -276,7 +301,13 @@ namespace l
|
||||
}
|
||||
|
||||
if(cfg->branches_mount_timeout > 0)
|
||||
l::wait_for_mount(cfg);
|
||||
{
|
||||
bool failure;
|
||||
|
||||
failure = l::wait_for_mount(cfg);
|
||||
if(failure)
|
||||
return 1;
|
||||
}
|
||||
|
||||
l::setup_resources(cfg->scheduling_priority);
|
||||
l::setup_signal_handlers();
|
||||
|
@ -421,7 +421,7 @@ check_for_mount_loop(Config::Write &cfg_,
|
||||
fs::PathVector branches;
|
||||
std::error_code ec;
|
||||
|
||||
mount = (std::string)cfg_->mountpoint;
|
||||
mount = *cfg_->mountpoint;
|
||||
branches = cfg_->branches->to_paths();
|
||||
for(const auto &branch : branches)
|
||||
{
|
||||
|
@ -16,6 +16,8 @@
|
||||
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
|
||||
*/
|
||||
|
||||
#include "to_string.hpp"
|
||||
|
||||
#include <cstdint>
|
||||
#include <string>
|
||||
|
||||
@ -55,4 +57,10 @@ namespace str
|
||||
{
|
||||
return s_;
|
||||
}
|
||||
|
||||
std::string
|
||||
to(const fs::Path &path_)
|
||||
{
|
||||
return path_.string();
|
||||
}
|
||||
}
|
||||
|
@ -18,6 +18,8 @@
|
||||
|
||||
#pragma once
|
||||
|
||||
#include "fs_path.hpp"
|
||||
|
||||
#include <cstdint>
|
||||
#include <string>
|
||||
|
||||
@ -28,4 +30,5 @@ namespace str
|
||||
std::string to(const int);
|
||||
std::string to(const uint64_t);
|
||||
std::string to(const std::string&);
|
||||
std::string to(const fs::Path&);
|
||||
}
|
||||
|
@ -64,6 +64,13 @@ public:
|
||||
return _data;
|
||||
}
|
||||
|
||||
const
|
||||
T&
|
||||
operator*() const
|
||||
{
|
||||
return _data;
|
||||
}
|
||||
|
||||
T*
|
||||
operator->()
|
||||
{
|
||||
|
Loading…
x
Reference in New Issue
Block a user