Because I wanted a complete backup of all my games, I decided to download them all into my 4TB drive. Because I use ZFS and want a more detailed snapshots of my games, I decided to split my library into multiple files.

Currently there are roughly 370 games installed, so doing this by hand would be a long task.

Analysis

But to automate this we have to take a look at the structure of the SteamLibrary fist

--SteamLibrary
  |--libraryfolder.vdf          // manifest for the library            
  |--steam.dll                  // DLL required by steam (Don't know why)
  |--steamapps                  // the actual library
     |--appmanifest_[0-9]*.acf  // manifest for each game
     |--common                  // the binares for each game
     |--downloading             // directory only used to download games/updates
     |--shadercache             // to store pre compiled shared to be used by games
     |--temp                    // for Temporary files?
     |--workshop                // everything installed from SteamWorkshop

Lets look a the libraryfolder.vdf first. This file

"libraryfolder"
{
        "contentid" "000"
        "label"     ""
        "launcher"  "C:\\Program Files (x86)\\Steam\\steam.exe"
}

This file is easy:

  • contentid: Is a number to identify this library within the steam app
  • label: A name given to make it easier for the user to identify the differentiate the libraries
  • launcher: Path to steam

steam.dll is a file we do not want to edit. It contains code needed by steam so we just leave it as is and copy it to our splitted libraries.

The next file we need to take a look at is appmanifest_[0-9]*.acf. Because this file is long (the shortest on my system was 40 lines) we take a look at one snipped.

appmanifest_948900.acf

"AppState"
{
        "appid"         "948900"
        "name"          "macdows 95"
        "installdir"            "macdows 95"
        // A lot of other parameters we don't care for this project
}

These three parameter are interesting for us:

  • appid: a numeric representation of the game within steam
  • name: the human visible name
  • installdir: the folder within common

With this we have all to move a single game from this library to a new one:

  1. Create target directory
  2. Create folder structure
  3. Copy steam.dll
  4. Create libraryfolder.vdf with unique contentid
  5. Copy/Move appmanifest_XXX.acf into steamapps
  6. Copy/Move games folder from common

Script the copy with linux

On my Debian - Linux system I did this:

APP_ID=[The steam app id, appmanifest_[appid].acf]
STEAM_SRC=[PATH TO YOUR SOURCE SteamLibrary parent folder]
STEAM_DST=[PATH TO YOUR DESTINATION SteamLibrary parent folder]
APP_NAME=$(find ${STEAM_SRC} -maxdepth 4 -type f -name appmanifest_${APP_ID}.acf | xargs grep 'name' | sed -E 's/.*"name"[[:space:]]+"(.*)"/\1/')
APP_DIR=$(find ${STEAM_SRC} -maxdepth 4 -type f -name appmanifest_${APP_ID}.acf | xargs grep 'installdir' | sed -E 's/.*"installdir"[[:space:]]+"(.*)"/\1/')

# If you want to create a partition, etc. for your new library. This line is where you should do this!

echo Start ${APP_NAME}
# Create base structure and copy dll and acf file
rsync -avz \
            --include="steam.dll" \
            --include="appmanifest_${APP_ID}.acf" \
            --exclude "*/common/*" \
            --exclude "*/workshop/*" \
            --include="*/"  \
            --exclude="*" \
            ${STEAM_SRC}/SteamLibrary/ ${STEAM_DST}/SteamLibrary
# Copy game data
rsync -avz \
            "${STEAM_SRC}/SteamLibrary/steamapps/common/${APP_DIR}/" \
            "${STEAM_DST}/SteamLibrary/steamapps/common/${APP_DIR}"
# Generate libraryfilder.vdf if it does not already
[ -f "${STEAM_DST}/SteamLibrary/libraryfolder.vdf" ] ||
   cat <<EOF >> "${STEAM_DST}/SteamLibrary/libraryfolder.vdf"
"libraryfolder"
{
        "contentid" "$(dd if=/dev/urandom bs=4k count=1 | grep -ao "[[:digit:]]" | tr -d '\n' | head -c 20)"
        "label"     ""
        "launcher"  "C:\\Program Files (x86)\\Steam\\steam.exe"
}
EOF

echo Finished ${APP_NAME}

Create seperate virtual disks for it

Ok, but i do not just want to extract the library, but also place it into an seperate zvol device.
This should also work btrf, etc, but the commands are different.

This is how I did it for my zfs installation

APP_ID=[The steam app id, appmanifest_[appid].acf]
STEAM_SRC=[PATH TO YOUR SOURCE SteamLibrary parent folder]
ZFS_BASE=[ZFS BASE PATH e.g. rpool/disks]

APP_SIZE_BYTES=$(du -cb "${STEAM_SRC}/SteamLibrary/steamapps/common/${APP_DIR}/" | tail -n 1 | cut -f 1)
# Buffer 100MB overhead for formatting, adjust if needed
APP_SIZE_WITH_BUFFER=$(( ${APP_SIZE_BYTES} + $(( 100 * 1024 * 1024 )) ))
TARGET_ZVOL=${ZFS_BASE}/steam-${APP_ID}-library

zfs create -V ${APP_SIZE_WITH_BUFFER} ${TARGET_ZVOL}
sfdisk /dev/zvol/${TARGET_ZVOL} <<EOF
label: gpt

type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7,
EOF
# Best for performance is to mtach cluster-size with zfs property volblocksize (16K is default)
mkfs.ntfs --cluster-size=16384 /dev/zvol/${TARGET_ZVOL}-part1
mkdir /mnt/steam-library-${APP_ID}
mount /dev/zvol/${TARGET_ZVOL}-part1 /mnt/steam-library-${APP_ID}
STEAM_DST=/mnt/steam-library-${APP_ID}

Let’s mount all the different disks

As earlier stated, windows only supports 24 (26-(“A”+”B”)) hard disks, we need to mount our hunderets of drive as folders. After all this trouble creating the single partitions for each app, lets mount them, or better add mount points and let windows do the heavy lifting.

        diskpart
        select volume // Automate
        assign mount="[Path where you want your libaray to be mounted. e.g. S:\Doki Doki Literature Club]"

Since Windows is limited to 24 hard disks, we need to mount our disks as folders.

And finally we need to add the (~370) libraries to steam

For this we need to edit the libraryfolders.vdf file at your steam installation directory. For me this is C:\Program Files (x86)\Steam\.

Again because this file is very long, lets look at a shortend version:

"libraryfolders"
{
        "0"
        {
                "path"          "C:\\Program Files (x86)\\Steam"
                // we ignore this part
        }
        "1"
        {
                "path"          "S:\\SteamLibrary"
                "label"         ""
                "contentid"             "1234"
                "totalsize"             "1234"
                "update_clean_bytes_tally"              "165689374333"
                "time_last_update_corruption"           "0"
                "apps"
                {
                        "10"            "292340389"
                        // and other apps
                }
        }
        "2"
        {
                "path"          "S:\\Doki Doki Literature Club - SteamLibrary" 
                "label"         "Doki Doki Literature Club"
                "contentid"             "4321"
                "totalsize"             "4321"
                "update_clean_bytes_tally"              "0"
                "time_last_update_corruption"           "0"
                "apps"
                {
                        // we ignore apps
                }
        }

After a quick look at this file we notice this pattern:

  • path: The parent directory of the SteamLibraray folder
  • label: A human readable name. The same we already defined at libraryfolder.vdf
  • contentid: The uniqe id defined in libraryfolder.vdf as contentid
  • The rest we can ignore safely! (Yes even the apps area)

So to add our artificial created libraries we just need to append this (templated) for each SteamLibrary to this file. Do not just copy this without using your brain!!!

"[Index, just +1 to the previous value]"
        {
                "path"          "[Parent directory of the library]"
                "label"         "[Same as in `libraryfolder.vdf`]"
                "contentid"     "[Same as in `libraryfolder.vdf`]"
                "apps"
                {
                        [No! We just ignore this!!!!]
                }
        }

Update

Having the data i analyzed how and which compression would be best: compare

Having 370+ different zvols the boot process was long since linux checkes all zvols for their partition tables (my zvols are on a HDD disk pool). Thats why i switched back to a single HDD. I tried to use virtiofs but the virtual file system is not compatible with EasyAntiCheat.