Skip to main content

GeeseFS

Last update:

GeeseFS is a program with which you can mount a bucket on a device or server and work with it as a regular folder. GeeseFS is suitable for working with a large number of objects, up to 1 MB each. You can work with GeeseFS on Linux, Windows and macOS.

Customize GeeseFS

  1. Configure access to S3.
  2. Install the client.
  3. Create a configuration.

1. Set up access to S3

Access can be configured by the Account Owner or a user with the role of iam.admin.

  1. Create a service user with a role with access to S3. If you use a service user with the role s3.user, object_storage_user or s3.bucket.user, an access policy must be configured in the bucket and its rules must allow access to this user.
  2. Issue an S3 key to the user.

2. Install the client

  1. Make sure that the FUSE utilities are installed by default:

    apt list --installed | grep fuse

  2. If the utilities are not installed, install them:

    sudo apt-get install fuse

  3. Download and install GeeseFS:

    wget https://github.com/yandex-cloud/geesefs/releases/latest/download/geesefs-linux-amd64
    chmod a+x geesefs-linux-amd64
    sudo cp geesefs-linux-amd64 /usr/bin/geesefs

  1. If you do not have WinFSP installed, download and install it.

  2. Download the geesefs-win-x64.exe file, but do not install it.

  3. Optional: rename the file to geesefs.exe for convenience.

  4. Create a geesefs folder.

  5. Move the geesefs.exe file to the geesefs folder.

  6. Add the geesefs folder to the PATH variable:

    6.1 In a Windows search, type Change System Environment Variables.

    6.2 Click Environment Variables.

    6.3. Select the PATH parameter.

    6.4. Click Edit → New.

    6.5 Add the folder path to the list.

    6.6. Press OK → OK.

  7. For the variable environment settings to be applied, reboot the device.

  1. Prepare and install the macFUSE package:

    1.1 Installing macFUSE requires tampering with macOS security settings, use the Getting Started tutorial on Github to do this.

    1.2 Install the macFUSE package.

  2. Open Terminal.

  3. Install GeeseFS:

    platform='arm64'
    if [[ $(uname -m) == 'x86_64' ]]; then platform='amd64'; fi
    wget https://github.com/yandex-cloud/geesefs/releases/latest/download/geesefs-mac-$platform
    chmod a+x geesefs-mac-$platform
    sudo cp geesefs-mac-$platform /usr/local/bin/geesefs

3. Create a configuration

  1. Open the CLI.

  2. Create a directory where the S3 key file will be stored:

    mkdir ~/.aws

  3. Create a credentials file:

    nano ~/.aws/credentials

  4. Add an S3 key:

    [default]
    aws_access_key_id = <access_key>
    aws_secret_access_key = <secret_key>

    Specify:

    • <access_key> - field value Access key from S3 key;
    • <secret_key> - field value Secret key from the S3 key.

  5. Press Ctrl + X → Y → Enter.

  1. Open PowerShell.

  2. Create a credentials file:

    New-Item -Type Directory -Path "$env:USERPROFILE\.aws" -Force | Out-Null
    notepad "$env:USERPROFILE\.aws\credentials"

  3. Add an S3 key:

    [default]
    aws_access_key_id = <access_key>
    aws_secret_access_key = <secret_key>

    Specify:

    • <access_key> - field value Access key from S3 key;
    • <secret_key> - field value Secret key from the S3 key.

  4. Click File → Save

  5. If the file is saved with a .txt extension, remove the extension:

    Rename-Item "$env:USERPROFILE\.aws\credentials.txt"
                "$env:USERPROFILE\.aws\credentials" -Force`

  1. Open Terminal.

  2. Create a directory where the S3 key file will be stored:

    mkdir ~/.aws

  3. Create a credentials file:

    cat ~/.aws/credentials

  4. Add an S3 key:

    [default]
    aws_access_key_id = <access_key>
    aws_secret_access_key = <secret_key>

    Specify:

    • <access_key> - field value Access key from S3 key;
    • <secret_key> - field value Secret key from the S3 key.

  5. Press Control + X → Y → Enter.

Mount the bucket

  1. Create a folder to mount:

    mkdir /mnt/<folder_name>

    Specify <folder_name> - the name of the folder to which the bucket will be mounted.

  2. Mount the bucket:

    sudo geesefs
    --endpoint https://<s3_domain>
    --region <pool>
    --profile <profile_name>
    <bucket_name> /mnt/<folder_name>

    Specify:

    • <s3_domain> - S3 API domain depending on the pool in which the bucket is located;
    • <pool> - pool where the buckets are located;
    • <bucket_name> - name of the bucket to be mounted;
    • <profile_name> - profile name, default default;
    • <folder_name> - name of the folder you created in step 1.

  3. Make sure the buckets are mounted:

    df -hT /mnt/<folder_name>

    Specify <folder_name> is the name of the folder you created in step 1.

  1. Open PowerShell.

  2. Mount the bucket:

    geesefs --endpoint <s3_domain> --region <pool>
    <bucket_name> <destination>

    Specify:

    • <s3_domain> - S3 API domain depending on the pool in which the bucket is located;

    • <pool> - pool where the buckets are located;

    • <bucket_name> - name of the bucket to be mounted;

    • <destination> - the name of the new disk or the path to the folder in Windows where the bucket will be mounted. To mount the bucket:

      • as a disk - use a value like S:;
      • as a folder - use the path to the folder in quotes, e.g. "C:\Data\GeeseFS".

Once mounted, the folder will be mounted in Finder as an external disk.

  1. Open the Finder or Terminal.

  2. Create a folder to mount:

    mkdir -p "<local_folder>"

    Specify <local_folder> - the path to the folder.

  3. Copy and save the path to the folder.

  4. Mount the bucket:

    geesefs -o volname="<volume_name>"
        --endpoint https://<s3_domain>
        --region <pool>
        --profile <profile_name>
        <bucket_name> "<local_folder>"

    Specify:

    • <volume_name> - name of the external disk in Finder, as which the mounted bucket will be displayed;
    • <bucket_name> - name of the bucket to be mounted;
    • <local_folder> - path to the folder you created in step 2;
    • <s3_domain> - S3 API domain depending on the pool in which the bucket is located;
    • <pool> - pool where the buckets are located;
    • <profile_name> - profile name, default default.

    A successful mount will return a response:

    main.INFO File system has been successfully mounted

  5. If you have a conflict due to a certificate, remove it:

    sed -i '' '/^[[:space:]]*ca_bundle[[:space:]]*=/d' ~/.aws/config

  1. Open the /etc/fstab file:

    sudo nano /etc/fstab

  2. Add a command to automatically mount the bucket:

    <bucket_name> /mnt/<folder_name> fuse.geesefs _netdev,allow_other,--endpoint=https://<s3_domain>,--region=<pool>,--profile=<profile_name> 0 0

    Specify:

    • <bucket_name> - name of the bucket to be mounted;
    • <folder_name> - name of the folder to which the bucket will be mounted;
    • <s3_domain> - S3 API domain depending on the pool in which the bucket is located;
    • <pool> - pool where the buckets are located;
    • <profile_name> - profile name, default default;

  3. Apply the settings:

    sudo systemctl daemon-reload

  4. Make sure the buckets are mounted:

    sudo mount -a
    df -hT

  1. Open PowerShell.

  2. Create a Windows service that will run with the OS:

    $dir = Join-Path $env:USERPROFILE '<folder_name>'
    New-Item -ItemType Directory -Path $dir -Force | Out-Null
    $script = Join-Path $dir '<task_name>.cmd'
    @"
    @echo off
    cd /d "%USERPROFILE%\\\<TASK_FOLDER>"
    geesefs.exe --endpoint https://<s3_domain> --region <pool> <bucket_name> <destination> >> "%USERPROFILE%\\\<task_name>\mount.log" 2>&1
    "@ | Set-Content -Path $script -Encoding ASCII

    schtasks /Create /TN "<task_name>" /SC ONLOGON /RL HIGHEST /RU $env:USERNAME /TR "$script" /F

    Specify:

    • <folder_name> - name of the folder in the profile, where you will store .cmd and logs;

    • <task_name> - name of the task that will mount the bucket;

    • <s3_domain> - S3 API domain depending on the pool in which the bucket is located;

    • <pool> - pool where the buckets are located;

    • <bucket_name> - name of the bucket to be mounted;

    • <destination> - the name of the new disk or the path to the folder in Windows where the bucket will be mounted. To mount the bucket:

      • as a disk - use a value like S:;
      • as a folder - use the path to the folder in quotes, e.g. " C:\Data\GeeseFS";

    • <task_name> - the name for the task in the Windows service that will start the mount bucket at system startup.

  3. Make sure the buckets are mounted:

    schtasks /Run /TN "<task_name>"

    Specify <task_name> is the name of the task in the Windows service that starts the mount bucket at system startup.

  4. Reboot the device.

  1. Open the Finder or Terminal.

  2. Create a folder to mount:

    mkdir -p "<local_folder>"

    Specify <local_folder> - the path to the folder.

  3. Copy and save the path to the folder.

  4. Copy the path to GeeseFS:

    which geesefs

  5. Create a configuration file for the autorun agent:

    mkdir -p "$HOME/Library/LaunchAgents"
    cat > "$HOME/Library/LaunchAgents/<agent_name>.plist" <<'PLIST'
    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
    <plist version="1.0">
    <dict>
    <key>Label</key>
    <string><agent_name></string>
    <key>ProgramArguments</key>
    <array>
        <string><geesefs_path></string>
        <string>-o</string><string>volname=<volume_name></string>
        <string>--endpoint</string><string><s3_domain></string>
    <string>--region</string><string><pool></string>
        <string>--profile</string><string><profile_name></string>
        <string><bucket_name></string>
    <string><local_folder></string>
    </array>
    <key>RunAtLoad</key><true/>
    <key>KeepAlive</key>
    <dict>
        <key>NetworkState</key><true/>
    </dict>
    <key>StandardOutPath</key><string>/Users/<username>/Library/Logs/geesefs.out.log</string>
    <key>StandardErrorPath</key><string>/Users/<username>/Library/Logs/geesefs.err.log</string>
    </dict>
    </plist>
    PLIST

    chmod 644 "$HOME/Library/LaunchAgents/<agent_name>.plist"
    chmod 700 "$HOME/Library/LaunchAgents"
    plutil -lint "$HOME/Library/LaunchAgents/<agent_name>.plist"

    Specify:

    • <agent_name> - file name for the autorun agent;
    • <geesefs_path> - path to the GeeseFS you copied in step 4;
    • <volume_name> - the name of the mounted bucket that will appear as an external disk in the Finder. May be the same as <local_folder> or different;
    • <s3_domain> - S3 API domain depending on the pool in which the bucket is located;
    • <pool> - pool where the buckets are located;
    • <local_folder> - path to the folder you created in step 2;
    • <profile_name> - AWS profile name, default default;
    • <bucket_name> - name of the bucket to be mounted to the device;
    • <username> - username on macOS.

  6. Start the agent:

    launchctl bootstrap gui/$(id -u) "$HOME/Library/LaunchAgents/<agent_name>.plist"
    launchctl enable gui/$(id -u)/<agent_name>
    launchctl kickstart -k gui/$(id -u)/<agent_name>

    Specify <agent_name> is the name of the autorun agent file you created in step 5;

  7. Make sure the buckets are mounted:

    mount | grep geesefs || echo "Not mounted yet"
    open "<local_folder>"

    Specify <local_folder> is the path to the folder you created in step 2.

Repair the tank

The method of unmounting depends on how the bucket was mounted - one-time or automatically.

Unmount the bucket:

sudo umount /mnt/<local_folder>

Specify <local_folder> - the path to the folder to which the bucket is mounted.

  1. Open PowerShell.

  2. Press Ctrl + C.

  3. Make sure the response appears on the command line:

    main.INFO Flushing all changes
    main.INFO Successfully exiting

  1. Open Terminal.

  2. Unmount the bucket:

    umount "<local_folder>" 2>/dev/null || diskutil unmount force "<local_folder>"

    Specify <local_folder> - the path to the folder to which the bucket is mounted.

  1. Check the mount list:

    systemctl list-units --type=mount

  2. In the row of the desired mount, copy the value of the UNIT column.

  3. Disable automatic mounting:

    sudo systemctl disable mnt-<folder_name>.mount

    Specify <folder_name> - the name of the folder to which the bucket was mounted.

  4. Make sure the buckets are unmounted:

    df -hT

  1. Open PowerShell.

  2. Uninstall the service with auto-mounted bucket:

    schtasks /Delete /TN "<task_name>" /F

    Specify <task_name> is the name of the task in the Windows service that starts the mount bucket at system startup.

  3. Reboot the device.

  4. Make sure that the buckets are no longer automatically mounted.

  1. Open Terminal.

  2. Check the mount point:

    mount | grep geesefs

  3. Copy the address of the mount point.

  4. Forcibly unmount the bucket:

    diskutil unmount force "<folder_path>"

    Specify <folder_path> - the path to the folder.

  5. Stop the mount agent and delete the configuration:

    launchctl bootout gui/$(id -u) "$HOME/Library/LaunchAgents/<agent_name>.plist"
    rm "$HOME/Library/LaunchAgents/<agent_name>.plist"

    Specify <agent_name> - the name of the autorun agent file.

  6. Make sure that the mounted bucket does not show up as a disk in Finder.