I used to think they needed to be this formal thing. I'd set up a Trello board, a Notion page, write a business plan in some cases.

But those projects ended up feeling too much like work for me to enjoy them, and I'd always end up ditching them as as result.

There's nothing wrong with ditching a side project, but ditching a side project that you invested "real world" expectations into feels kinda shitty.

After all, what's the point of a side project?

That answer is probably a little bit different for everyone; it could be something for your portfolio. It could be to test out a new technology or framework. It could be something that you're looking to make money from.

Or it could just be for fun.

No matter the case, setting expectations for the project is key. The expectations should align with the overall goal of the project.

Most side projects should come with the expectation that they will be abandoned; the only exceptions are side projects for a portfolio when trying to get hired.

Side projects you're looking to make money from are not side projects. They are another class of application altogether, and thinking of them as side projects is detrimental.

Design your side project around the reason you're making it, not around what the project actually is. If you're making a movie app so you can learn Svelte -- focus more on how you can push your limits with Svelte, and less trying to make the best movie app.

Don't be afraid to take your time and explore. Take detours and scenic routes. Make U-turns with abandon. Get lost.

This is the very essence of the side project.

In An intro to Minecraft mods, I outline my reasons for choosing the Fabric framework over Forge. Continuing with that same reasoning, this post will be outlining multiple approaches for running a Minecraft server using Fabric (although at a high level, the concepts should still apply to Forge).

You may want to do some research beforehand to determine which mods you want to run on your server, since Fabric mods will only run on a Fabric server, and Forge mods will only run on a Forge server.

If you're not sure what a Minecraft server is, you can simply think of it as a place to house Minecraft mods. To get a better idea of how a server fits into modding in general, read the Fabric Server section of the aforementioned post.

Overview

You might be wondering where the "multiple approaches" bit comes in, since we are only focusing on Fabric. We'll be covering the following ways to host a Fabric server:

• local machine

  • Windows

  • Linux/MacOS

• VPS

Regardless of where the server is hosted, it can be invoked (or "spun up") in multiple ways:

• raw commands in a shell

• running a script file

• running the server in a Docker container

Choosing a server location

In the Development Environment section of the first post, we covered the fact that your development environment matters, in terms of which computer you use. Conversely, the computer you choose to use as your server doesn't matter at all -- so long as it has the proper version of Java and enough RAM to run Minecraft.

Separate machine vs local machine

Speaking of RAM, that is one reason you might choose to run your server on a separate computer; the default amount of RAM allocated to the Minecraft server is 2GB, which your computer may not have to spare.

Even if RAM isn't an issue for you, you may still want a separate machine for other reasons -- a big one being availability; if you want your server to be accessible round-the-clock, you likely want a separate machine. Otherwise, your server goes down if your computer shuts off or loses network connectivity.

On the other hand, if your server needs are more casual and constant availability isn't a concern, then running the server locally (from your main machine) might be enough for you.

A new computer isn't necessary

You also might be thinking, "I can't afford to buy another computer to run a server on". If that's the case, you could explore these options:

• using an old computer

• buying a Raspberry Pi (can be found for less than $100)

• using a Virtual Private Server (monthly cost depending on usage and server specs)

Docker

Docker might be a good option if you want to be able to reuse the same server configuration on multiple devices, or if you don't want to alter an existing Java configuration. Here are a few things to note about running a server in a Docker container:

• it will still consume RAM on whatever computer is running the container

• no Java configuration is required

  • Java is automatically installed in the container

• Docker containers don't "care" where they are run from

  • you can bundle the container's configuration into a file on your local machine

  • that same file can be used to run an identical container on a separate machine (regardless of that machine's OS)

• Docker can run from

  • your local machine

  • another physical machine

    • an old laptop

    • a Raspberry Pi

  • a Virtual Private Server

  • ...or any other hardware that supports Docker.

Here's an example of when you might want to use Docker:

You are testing out a Minecraft server configuration with multiple mods, and multiple custom game settings. You're testing this configuration out on your laptop, but you want to be able to recreate the configuration on another machine without having to install each mod and enable each custom game setting all over again.

This is a great reason to use Docker. At a high level, the configuration would look something like this:

• find and download a Docker image that is preconfigured for running a Minecraft server

  • think of the image as the blueprint for the container

• use the image to run a container

  • the image's blueprint is used to construct and "run" the container

  • think of a container as a self-contained environment for which to run software

• load the mods you want to test into the container via a volume

  • a volume is a way to link file storage between the host machine and the container

    • the host machine is the computer that Docker is installed on

  • the volume is where the /mods folder will live

• once you've decided on the game settings and mods you want to use, put instructions on how the container should be run in a docker-compose.yml file

  • Docker Compose files describe the specifics of how the server container be run, such as

    • the version of Minecraft to use

    • which user(s) to grant operator permissions

    • game settings (max players, spawn point, mobs, etc.)

We'll illustrate these concepts with examples later in the Running a Server with Docker section.

Running a Server

Java Prerequisite

If you plan on running a server on a machine other than your own, make sure to install Java on that machine, following the instructions provided by Fabric.

If you're using Docker, make sure the image you're using has the correct version of Java. The image likely has a way to specify a Java version with a tag for the image.

If you're using Linux for your server, the instructions Fabric provides for setting up Java are fairly basic. For more detailed instructions, see Installing Java on Ubuntu.

Loading a mod onto the server

For mods to work on the server, we need to make sure that our Minecraft client has those same mods. More specifically, that means:

• you have an installation of the Fabric launcher (client) whose version matches the Minecraft version of your server

  • this is the launcher you select when you open the Minecraft application on your computer

• the /mods folder for your Fabric launcher installation contains:

  • the .jar file for the proper version of the Fabric API

  • a .jar file for each mod

• the /mods folder for your Fabric server must contain the same .jar files as the /mods folder in the Fabric launcher

In other words, the /mods folder within your Fabric launcher client should have the same contents as the /mods folder on your Fabric server. If you want to read more about the reasoning, check out the Fabric docs on this topic.

• once the /mods folders match on the client and server, launch the server

• once the server is running, grant op (operator) permissions to necessary player(s)

• connect to the server using the Connecting to your server instructions below

• verify that you can use the mod

Approach-specific instructions are detailed below.

Running a Server on Linux/MacOS

This is perhaps the most accessible option since it can be run from:

• a machine running Linux

  • Raspberry Pi

  • VPS

• a machine running MacOS

Here we'll follow instructions from the Fabric wiki, and walk through them as we go.

If you've been following along, you should already have Java installed. The next step is to go to the server download page and download the relevant version:

# make a directory for the server
mkdir fabric-server
cd fabric-server

# download the file to our fabric-server directory
curl -OJ https://meta.fabricmc.net/v2/versions/loader/1.19.4/0.14.19/0.11.2/server/jar

# confirm the download worked
ls
# output
fabric-server-mc.1.19.4-loader.0.14.19-launcher.0.11.2.jar

Now that we have the server's .jar file downloaded, we can use the command supplied on the download page to launch the server using that .jar file:

# launch the server, allocating 2GB RAM
java -Xmx2G -jar fabric-server-mc.1.19.4-loader.0.14.19-launcher.0.11.2.jar nogui

We get an error, which is expected because we need to accept the EULA:

[21:10:21] [main/ERROR]: Failed to load properties from file: server.properties
[21:10:21] [main/WARN]: Failed to load eula.txt
[21:10:21] [main/INFO]: You need to agree to the EULA in order to run the server. Go to eula.txt for more info.

If we inspect the contents of our server folder, we can see that files and directories have been generated as a result of attempting running the server. Among these is the file eula.txt:

# in fabric-server directory
ls -la # press enter

# output
drwxr-xr-x  4 idev idev   4096 Jun 12 21:10 .fabric
-rw-r--r--  1 idev idev    158 Jun 12 21:10 eula.txt
-rw-r--r--  1 idev idev 155095 Jun 12 21:09 fabric-server-mc.1.19.4-loader.0.14.19-launcher.0.11.2.jar
drwxr-xr-x  8 idev idev   4096 Jun 12 21:10 libraries
drwxr-xr-x  2 idev idev   4096 Jun 12 21:10 logs
drwxr-xr-x  2 idev idev   4096 Jun 12 21:10 mods
-rw-r--r--  1 idev idev   1272 Jun 12 21:10 server.properties
drwxr-xr-x  3 idev idev   4096 Jun 12 21:10 versions

Now we can open that file with our preferred text editor and change false to true:

vim eula.txt

# inside eula.txt
eula=true
# save and exit eula.txt

Now that we've agreed to the EULA, we can re-run the command to launch the server:

# launch the server, allocating 2GB RAM
java -Xmx2G -jar fabric-server-mc.1.19.4-loader.0.14.19-launcher.0.11.2.jar nogui

The server starts up this time with no errors:

[21:15:16] [Worker-Main-2/INFO]: Preparing spawn area: 91%
[21:15:16] [Worker-Main-2/INFO]: Preparing spawn area: 94%
[21:15:17] [Worker-Main-5/INFO]: Preparing spawn area: 97%
[21:15:17] [Server thread/INFO]: Time elapsed: 28858 ms
[21:15:17] [Server thread/INFO]: Done (35.063s)! For help, type "help"

Now that the server is running, we can click into the terminal and type op your_player_name

[14:24:08] [Server thread/INFO]: Time elapsed: 8357 ms
[14:24:08] [Server thread/INFO]: Done (10.491s)! For help, type "help"
op awesomeminecraftplayer42023
[14:25:01] [Server thread/INFO]: Made maddoxchunk a server operator

Running a Server on Windows

The Fabric instructions do a pretty good job of walking through this one, but we'll go through the steps here anyway (starting with STEP 3: Install Fabric in the Server Folder):

• go here and click the 'download for Windows' button

• run the file

• select the 'Server' tab

• select the version and install the location

  • the Minecraft version should match the Minecraft version of the mod(s) you want to use

  • note that this file location has nothing to do with your installation of Minecraft

• press 'Install'

• click 'Download server jar'

• click 'Generate'

• open the folder location where you saved the server to

• run the start.bat file

• you'll get an error about the EULA

• close the terminal

• open eula.txt in the server folder that has the start.bat file

• change false to true in eula.txt

• save and exit the file

• run start.bat again

• click into the terminal with the server running and type op your_player_name to grant operator permissions

Running a Server with Docker

I won't go over installing Docker, since their official docs should be enough to get you there.

It's worth mentioning that once you have Docker installed, the steps below will work the same (more or less), regardless of your operating system.

Run a Vanilla Minecraft server in a container

Once you've got Docker installed, it's time to download and run a Minecraft server image.

• start by going to the documentation for the Docker image we're using

• run the listed command

  • docker run -d -it -p 25565:25565 -e EULA=TRUE itzg/minecraft-server

  • this runs the container in the background so that our terminal is free to run other commands

  • confirm the container is running with docker ps

• copy the CONTAINER ID from the docker ps command

• run docker stop <CONTAINER ID> to stop the container

• run docker ps again to confirm the container is not running

Run a Fabric Minecraft server in a container

The docker command above allowed us to run a "vanilla" Minecraft server. However, we are interested in running a Fabric server, so we consult the Types and platforms documentation.

From here we can select 'Fabric' from the menu on the left. We can see a command that instructs us on how to run a Fabric server by using the -e TYPE=FABRIC flag. However, there is something else that's different about this command: it's mounting a volume with -v /path/on/host:/data. Before we run this command, we need to set up that directory on our host machine (the one running Docker).

Mounting a data directory

If we head to the Data directory documentation, we get some instructions on how to do this.

Let's try running a container with an attached data volume, using the supplied command from the Fabric section of the image docs:

mkdir mc-data

docker run -d -v ./mc-data:/data \
    -e TYPE=FABRIC \
    -p 25565:25565 -e EULA=TRUE --name mc itzg/minecraft-server

Okay, so we've started a container and mounted the volume, but how do we use it?

It's a bit difficult to picture right now, since we're running the container as a background process with the -d flag. Let's try running it again without that flag and see what happens:

# first, remove/stop the container
docker rm mc

# run the same command without the -d flag
docker run -v ./mc-data:/data \
    -e TYPE=FABRIC \
    -p 25565:25565 -e EULA=TRUE --name mc itzg/minecraft-server

# output
[init] Running as uid=1000 gid=1000 with /data as 'drwxrwxr-x 2 1000 1000 4096 Jun 15 11:12 /data'
... # (more output)
[init] ERROR: version lookup failed:

This isn't an error with the volume. We would get the same error if we ran the command without the -v flag. The only reason we didn't see the error before is that we ran the process in the background, so the output wasn't visible.

This error is because we need to specify a Fabric/Minecraft version for the container to run.

Let's try it again, this time using the appropriate flags to specify the version:

docker rm mc

docker run -v ./mc-data:/data \
    -e TYPE=FABRIC \
    -e VERSION=1.19.4 \
    -p 25565:25565 -e EULA=TRUE --name mc itzg/minecraft-server

We still get an error, but this time it's different:

[init] ERROR failed to install Fabric launcher from 1.19.4, LATEST, LATEST

This is yet another misleading error message. In my case, it was happening due to an issue with WSL2. The issue is resolved by deleting the ~/.docker/config.json file.

If you encountered the error above, go ahead and delete the config.json file, then run the commands again to make sure it works:

docker rm mc

docker run -v ./mc-data:/data \
    -e TYPE=FABRIC \
    -e VERSION=1.19.4 \
    -p 25565:25565 -e EULA=TRUE --name mc itzg/minecraft-server

Now that we know we can get the server running, it's time to add our mod.

Loading mods onto the server

Looking again at the Data directory documentation, we can see that the /mods folder lives inside the /data directory, which we have created as mc-data on our host machine.

Speaking of the data directory, if we inspect the contents of mc-data, we can see that the folder is no longer empty since we have successfully run the server:

# stop the server if it's still running using CTRL + C
ls mc-data

#output
banned-ips.json      fabric-server-mc.1.19.4-loader.0.14.21-launcher.0.11.2.jar  ops.json           whitelist.json
banned-players.json  libraries                                                   server.properties  world
crash-reports        logs                                                        usercache.json
eula.txt             mods                                                        versions

Recall from Loading a mod onto the server that the /mods folder on our server should match the /mods folder in our Fabric launcher installation. As such, we copy the contents of that folder to our data directory for the container

# copy the contents of the Fabric 
# launcher's /mods folder into the mc-data/mods folder
cp -a /path/to/fabric/launcher/mods .mc-data/mods

Now we are ready to run our server again:

docker rm mc

docker run -v ./mc-data:/data \
    -e TYPE=FABRIC \
    -e VERSION=1.19.4 \
    -p 25565:25565 -e EULA=TRUE --name mc itzg/minecraft-server

Go ahead and connect to your server using the instructions below.

The game loads up when we connect, but we have a problem -- we aren't able to run commands. Normally, we could just go to our terminal and op our player, but things work a bit differently with Docker. Since we are running the server in a container, we don't have direct access to the terminal like we do when we run the server directly in the operating system.

So how do we issue commands to our Docker container? Enter RCON.

RCON

RCON is short for Remote Control, and allows Minecraft commands to be issued remotely.

Okay, but... how do we use it?

Thankfully, the Docker image we're using includes rcon-cli. Let's see it in action:

# stop the container (if running)
docker stop mc

# start the container again
docker start mc

Before we can run rcon-cli, we need to wait for server to completely load, since RCON does not become enabled until that point. When you see the following message in the terminal, you're ready to use rcon-cli:

[03:28:23] [Server thread/INFO]: Done (8.096s)! For help, type "help"
[03:28:23] [Server thread/INFO]: Starting remote control listener
[03:28:23] [Server thread/INFO]: Thread RCON Listener started
[03:28:23] [Server thread/INFO]: RCON running on 0.0.0.0:25575

Once the RCON service has started, we can open a new terminal window and use rcon-cli:

# replace your_player_name with the actual name of your player
docker exec mc rcon-cli op your_player_name

# output
Made your_player_name a server operator

Now that you are an operator, you can use commands in Minecraft. Go ahead and use the appropriate command(s) to test out your mod (if applicable).

Docker Compose

When I originally explained the use cases for Docker, a big selling point was the portability, and the ability to reuse configurations. So far, we haven't seen how Docker can achieve that. We ran a bunch of commands, and even hit some errors in the process.

How can a workflow like that be reusable?

The answer is that it can't, and that's where Docker Compose comes into play.

Now that we have a working configuration, we can package that configuration into a docker-compose.yml file so that we don't have to remember the commands. Instead, we can just point Docker to the docker-compose.yml file.

Let's go through the instructions:

• first install Docker Compose by following the instructions here.

• make a new directory and change into it

    mkdir mc-compose 
    cd mc-compose
  

• create a docker-compose.yml file and put the following in it

note that the file below is slightly different from the file linked in the documentation (we added the VERSION and TYPE environment variables)

also note that indentation matters in a .yml file

•       version: "3.8"
  
        services:
          mc:
            image: itzg/minecraft-server
            tty: true
            stdin_open: true
            ports:
              - "25565:25565"
            environment:
              EULA: "TRUE"
              VERSION: "1.19.4"
              TYPE: "FABRIC"
            volumes:
              # attach the relative directory 'data' to the container's /data path
              - ./data:/data
  

Using an .env file

To make things even more reusable, we can use an .env file located in the same folder as our docker-compose.yml file.

vim .env

# .env
VERSION=1.19.4
SERVER_TYPE=FABRIC
DATA_DIR=./mc-data

Now we can update our docker-compose.yml:

# docker-compose.yml
version: "3.8"

services:
  mc:
    image: itzg/minecraft-server
    container_name: mc
    tty: true
    stdin_open: true
    ports:
      - "25565:25565"
    environment:
      EULA: "TRUE"
      VERSION: "${VERSION}"
      TYPE: "${SERVER_TYPE}"
    volumes:
      # attach the relative directory 'data' to the container's /data path
      - "${DATA_DIR}:/data"

Run the server with docker compose up to make sure the new config works.

Automating RCON

Remember how we had to wait for the server to spin up before we could use rcon-cli? That was kind of annoying. It would be nice if there were a way to automate this.

Thankfully, someone made a pull request, and some new commands were added to the image. These commands allow rcon-cli to run automatically, either when the server starts, or when the game client connects to the server.

We'll keep things simple and just automate the provisioning of operator permissions. In the docker-compose.yml file below, you'll notice a new RCON_CMDS_STARTUP property under environment:

# docker-compose.yml
version: "3.8"

services:
  mc:
    image: itzg/minecraft-server
    container_name: mc
    tty: true
    stdin_open: true
    ports:
      - "25565:25565"
    environment:
      EULA: "TRUE"
      VERSION: "${VERSION}"
      TYPE: "${SERVER_TYPE}"
      RCON_CMDS_STARTUP:  |-
        /op your_username_goes_here
    volumes:
      # attach the relative directory 'data' to the container's /data path
      - "${DATA_DIR}:/data"

Go ahead and run the container with docker compose up. If you get an error like Additional property RCON_CMDS_STARTUP is not allowed, the issue is likely that something isn't indented properly. Go back into the file and make sure the new command is indented one level in from environment.

But remember, we want this docker-compose.yml file to be as reusable as possible. Once it's complete, any changes that we need to make to the configuration should be done via the .env file. Let's put our player name into the .env file:

vim .env

# .env
VERSION=1.19.4
SERVER_TYPE=FABRIC
DATA_DIR=/mnt/c/Users/iDev/fabric-1.19.4-client/
OP_PLAYER=your_username_goes_here

Now we can adjust our docker-compose.yml file accordingly:

# docker-compose.yml
version: "3.8"

services:
  mc:
    image: itzg/minecraft-server
    container_name: mc
    tty: true
    stdin_open: true
    ports:
      - "25565:25565"
    environment:
      EULA: "TRUE"
      VERSION: "${VERSION}"
      TYPE: "${SERVER_TYPE}"
      RCON_CMDS_STARTUP:  |-
        /op ${OP_PLAYER} # note that there are no quotes in this command
    volumes:
      # attach the relative directory 'data' to the container's /data path
      - "${DATA_DIR}:/data"

Note that the syntax is slightly different for this command, compared to the others. There are no quotes around the command. Using quotes will cause an error.

It's also possible to specify more than one player, but the error handling still needs to be worked out. This means that if you make a typo in one player's name, or one of the players doesn't exist, the entire command will fail.

If you're interested in experimenting with this, the syntax for specifying multiple players is as follows:

# .env
# OP_PLAYERS=player1:player2:player3 # this will fail; no valid players
OP_PLAYERS=your_user_name_goes_here:player2 # this will fail; at least one invalid player

Reusing a Docker config

So far, we've seen how using Docker Compose can save us from having to type long complicated commands. We can take it a step further and place our docker-compose.yml file on another machine so that it can run the same container with the same configuration.

First things first, we'll copy the file from our local machine over to a Virtual Private Server that has Docker installed. Also note that a user named mc-admin has already been created on the VPS:

# in mc-compose folder on local machine
# make sure to add the '/' at the end of vps-mc!
rsync -a ./docker-compose.yml digitalocean.minecraft:/home/mc-admin/vps-mc/

If you're unfamiliar with rsync, here is what the command above does:

• takes the docker-compose.yml file in our current directory on the local machine

• uses SSH behind the scenes to log into the VPS (digitalocean.minecraft)

  • an SSH config file obscures the username and IP of the Digital Ocean VPS we're logging into

  • to execute without a SSH config file:

    • replace digitalocean.minecraft with user@ip.address.of.vps

• creates a directory vps-mc/ on the VPS, under the existing mc-admin user's /home directory

  • this is why the / is important at the end of the command

  • without the /, vps-mc will be created as a file instead of a folder

• places the docker-compose.yml file in the /home/mc-admin/vps-mc directory of the VPS

Since I've previously configured ssh, I don't need to supply a username and password when I run the command above.

If you haven't configured ssh, it would look more like this:

# we need to specify username@server
rsync -a  ./docker-compose.yml mc-admin@docker.vps.ip:/home/mc-admin/vps-mc/

If you want to learn how to simply your ssh flow like I have, make sure to check out my Managing SSH Keys post.

Before we run the server, we need to transfer over a few more files and folders:

rsync ./.env digitalocean.minecraft:/home/mc-admin/vps-mc/
rsync -a --mkpath ./mc-data/mods/ digitalocean.minecraft:/home/mc-admin/vps-mc/mc-data/mods/

Now we're ready to run the server on the VPS:

ssh digitalocean.minecraft

# now in vps
cd vps-mc
docker compose up

# output
permission denied while trying to connect to the Docker daemon socket at 
unix:///var/run/docker.sock: 
Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.config-hash%22%3Atrue%2C%22com.docker.compose.project%3Dpi-mc%22%3Atrue%7D%7D": 
dial unix /var/run/docker.sock: connect: permission denied

Okay, so we got a permissions error. No biggie, this is a pretty common thing to run into with Docker. I had already configured it on my local machine, but we need to set it up on the VPS.

Enabling rootless Docker

Yes, prefixing Docker Compose command with sudo would fix our issue, but running Docker as root goes against best practices, so we're going to do it the proper way:

You can follow along with this step by viewing the Docker documentation.

• first we'll start by updating our package manager

  • sudo apt-get update && sudo apt-get upgrade

• next we'll install the package that Docker tells us we need

  • sudo apt-get install uidmap

• follow the instructions under Distribution-specific hint

  • sudo apt-get install -y dbus-user-session

  • the package is already installed, so no need to log out/back in

• follow the instructions under the Install section

  • dockerd-rootless-setuptool.sh install

  • receive command not found error

  • continue following instructions: sudo apt-get install -y docker-ce-rootless-extras

  • received another error: Unable to locate package docker-ce-rootless-extras

  • switch to the Without Packages tab and run curl -fsSL https://get.docker.com/rootless | sh

The installation was successful, but we have some output to read through:

# output from the install script
[INFO] Installed docker.service successfully.
[INFO] To control docker.service, run: `systemctl --user (start|stop|restart) docker.service`
[INFO] To run docker.service on system startup, run: `sudo loginctl enable-linger sharif`

[INFO] Creating CLI context "rootless"
Successfully created context "rootless"
[INFO] Using CLI context "rootless"
Current context is now "rootless"

[INFO] Make sure the following environment variable(s) are set (or add them to ~/.bashrc):
export PATH=/usr/bin:$PATH

[INFO] Some applications may require the following environment variable too:
export DOCKER_HOST=unix:///run/user/1000/docker.sock

Going through the sections of output one by one:

• since I want Docker to run when the system starts, I run the provided command

  • sudo loginctl enable-linger sharif

• I need to set both the PATH and DOCKER_HOST environment variables

  • I'm using zsh and not bash, so environment variables are set differently

  • I refer to a previous blog post of mine to set the environment variables at the profile level

Now we should be able to run docker compose up from our vps-mc directory again without getting a permissions error.

Connecting to your server

On our computer with Minecraft installed, we can open our Minecraft Launcher, and confirm that we can connect to the server.

• select the Fabric Launcher installation

  • make sure the version of Minecraft for the launcher aligns with the version of Minecraft for your server

• when the game loads, select 'Multiplayer'

• accept the warning about third-party online play

• select 'Direct Connection'

• depending on how you set up your server, you'll need to enter the hostname accordingly

  • local server: localhost:25565

  • VPS: <IP of your server>:25565

• click 'Join Server'

Resources

• Install Docker in WSL2 without Docker Desktop

• Run Docker as a non-root user

• Docker volumes documentation

• Docker .env file documentation

• Minecraft server Docker image documentation

• Fabric server .jar download (Linux/MacOS)

• Fabric server installer for Windows

• Fabric client/server patterns

• Install Java on Windows

• Installing Java on Ubuntu

• An Intro to Minecraft Mods

I started looking into what was involved in coding a mod for Minecraft, and found it to be more involved than I was expecting.

I figured there had to be others in the same boat, so I decided to document what I learned along the way.

Hope this helps!

What are we covering?

• installing Java

• enabling mods in Minecraft

• Minecraft servers

• setting up VS Code for Java development

• making a basic mod with Fabric

• testing our mod in single-player mode

• installing a mod on a server

• troubleshooting

• theory

What do I need?

Once you start poking around online, it's easy to get overwhelmed with all of the resources on the topic of Minecraft mods.

There are multiple versions of Minecraft, multiple releases of those versions, multiple programming languages that can be used for mod development, and multiple modding frameworks within those language options.

There's also a lot of outdated information, as well as information that only relates to older versions of Minecraft. I want to be able to make mods for the newest version of Minecraft, and so that's what I outline in this blog post.

Minecraft Java Edition

The Java Edition of Minecraft is synonymous with the "computer version" of Minecraft. If you have Minecraft installed on Windows, MacOS, or some desktop variation of Linux, then you have the Java Edition.

In other words, you need a computer that has Minecraft installed. This is the computer that you'll be using to develop the mod.

A Minecraft server

We get into the role of a Minecraft server later in this blog post, and in even more detail in Running a Minecraft Server. For now, just think of a Minecraft server as a place for mods to live.

If you're not planning on playing multiplayer, then you don't need to worry about this step much, since the only server you'll need is a development server (which comes bundled with the Fabric framework).

If you want to learn more about servers, check out Running a Minecraft Server.

A programming language

At the top of this section, I mentioned that Minecraft can be modified with multiple programming languages. Depending on where you're looking, you might come across resources for:

• Java

• Python

• JavaScript

I initially wanted to use JavaScript because that's what I'm most familiar with. But, after some experimentation and research, I decided JavaScript was not worth it for this use case.

Why? Because it adds a layer of complexity. Even if you're coding the mod in JavaScript (or TypeScript), it's getting converted to Java at the end of the day. So while it might be a more ergonomic experience to use JavaScript, the setup and dependency overhead feels less than ideal to me.

To keep things as simple and direct as possible, we're going to code the mod with Java.

A way to modify Minecraft

Now we start to get a bit more technical. We know we need a server to house the mods, but how do we make the mods?

For that, we need something that hooks into the existing Minecraft code and allows us to make changes. Modifying the Minecraft code directly doesn't seem like a great idea. After all, we could (and probably would) break something.

Thankfully, some smart people worked through this problem and came up with the idea of a modification framework. Frameworks provide a safer and easier way to make modifications when compared to modifying source code directly.

We get into the details later in this blog post. For now, the important thing to know is that frameworks package your mod into a .jar file, which lives on your Minecraft server (or within your local installation of Minecraft).

.jar files also provide modularity -- each mod exists in its own .jar file. If you decide you don't want a certain mod anymore, its .jar file can simply be deleted from the server.

For Java, there are two frameworks to choose from:

• Forge

• Fabric

I will briefly compare the two, then explain why I chose Fabric.

Forge

Forge is the original framework, and the more powerful of the two. As such, it can handle more drastic modifications compared to Fabric. Additionally, Forge is better at handling "interop", meaning it's easier for mods to "talk to" other mods.

This all sounds great, but it comes with some tradeoffs:

Because it is more powerful and flexible, that also makes it more demanding of your machine's resources. If your machine isn't built for performance, it may not be able to handle Forge.

Forge's power and flexibility come at the cost of a larger, more complex codebase. More complexity and more code mean longer release cycles -- much longer. In fact, many mods created with Forge are still out-of-date because updating them to comply with new Minecraft versions simply takes too long.

Fabric

Fabric was created out of the need for a more lightweight, performant, and maintainable framework. It can be used on a standard laptop, and mods built with Fabric can be kept up-to-date as a result of Fabric's lighter footprint.

Of course, tradeoffs work both ways. Because it is more lightweight, the modification capabilities are not as robust as those of Forge, and so the "interop" of Fabric mods suffers; this means it's harder for Fabric mods to talk to each other when compared to Forge.

I decided to use Fabric because:

• I will sometimes be working "on the go", so I need to be able to develop the mod with my standard laptop

• I don't need to make complex mods that work together

• I want to work with the latest version of Minecraft

If you decide Forge is better for what you need, I wish you the best of luck. The rest of this guide will focus on Fabric.

Setup

You probably already have Minecraft installed on your computer, so the next step is to give Minecraft the ability to "run" mods.

It seems that there is a lot of outdated content on this topic, and more recent releases of Minecraft (1.19.4 at the time of this writing) have different setup and configuration instructions when compared to the older releases.

Thankfully, Fabric has a wiki.

Fabric Launcher (Client)

Per the wiki, we first need to install the Fabric launcher. Before that, though -- we should address what exactly the Fabric Launcher is. You can think of the Fabric Launcher as a special installation of Minecraft that allows mods to be loaded into the game. If you've ever installed a different version of Minecraft under the 'Installations' tab, it's very similar to that.

As a prerequisite, we need to make sure Java 17 is installed on the computer that Minecraft Java Edition is installed. You most likely have Minecraft installed on Windows or MacOS, but if you have it installed on Linux, you might need some additional instructions, as the Linux instructions on the wiki don't cover certain edge cases.

It's also worth noting that the commands in Linux CLI instructions are for older versions of Minecraft. If you copy the command from these instructions, it will install Java 8 (we need Java 16+). For more detailed instructions on installing Java on Linux via the CLI, see Installing Java on Ubuntu.

I'm on Windows, so I follow the appropriate instructions to verify my Java Installation.

Following the instructions, I check my Java installation in the Windows Command Prompt:

java -version # press enter

# output
java version "19.0.2" 2023-01-17
Java(TM) SE Runtime Environment (build 19.0.2+7-44)
Java HotSpot(TM) 64-Bit Server VM (build 19.0.2+7-44, mixed mode, sharing)

I see a couple of issues:

• The Java version is 19 and not 17

• I have the JRE (Java Runtime Environment), but not the JDK (Java Development Kit)

I follow the link on the page to the Java 17 installer (Windows).

Per the instructions, I also select the options to set JAVA_HOME and the Oracle registry keys:

Once the installation completes, I close the terminal, open a new one, and proceed to check the Java installation once more:

java -version # press enter

# output
openjdk version "17.0.7" 2023-04-18
OpenJDK Runtime Environment Temurin-17.0.7+7 (build 17.0.7+7)
OpenJDK 64-Bit Server VM Temurin-17.0.7+7 (build 17.0.7+7, mixed mode, sharing)

Everything reads as version 17 (including the JRE), so I move on to the next step of installing the launcher, again following the instructions:

• download the Fabric Installer .exe file

• run the .exe file and choose the latest version (1.19.4)

  • keep all other defaults

• create a new folder for the launcher installation: C:\myUser\fabric-minecraft

• open Minecraft Launcher

• go to the 'Installations' tab

• find the launcher installation

  • press the '...' button

  • select 'Edit'

  • change the 'Game Directory' from the default location to the folder from the step above

  • press 'Save'

• back on the main screen, press 'Play' so that Minecraft installs in that folder

  • this step will also add the /mods folder to the folder you selected for installation

• close Minecraft

• download the Fabric API

• place the Fabric API .jar file in C:\myUser\fabric-minecraft\mods

• download the Mod Menu mod (so we can make sure mods can load)

• place it in the \fabric-minecraft\mods folder

• open Minecraft and confirm you now have the 'Mods' menu on the home screen

So, what did we just do?

We just created a custom version of Minecraft that enables modifications (mods).

The Fabric API itself is a mod. We can think of it as the "parent mod", "controller mod", etc. -- basically, we need the Fabric API mod for the other mods to run against. When you drop a mod into the /mods folder, it must be a mod that is supported by the Fabric API (or one you developed using the Fabric API).

Fabric Server

At this point, you might wonder what we need a server for. We already have a version of Minecraft where we can install mods, so what role does the server play?

One use case for a server is to play multiplayer with mods. Think about it -- let's say we download and install a few more mods, and we want to use those mods to play with friends online.

Right now, we don't have a way of doing that without joining an existing server that someone else created. When you play Single Player mode, it uses the Minecraft code (along with any mods) that lives on your computer's hard drive.

To play Multiplayer with mods, we need to load the Minecraft code from a hard drive that our friends can access, too -- a hard drive that is accessible either via a local or public network.

And for that, we need a server.

A Fabric server takes the same concept as the Fabric Launcher and puts it into the context of a server. This way, your friends can connect to that server and enjoy the mods that you've installed in the server's /mods folder.

But... wait a second, we're interested in making mods, not playing multiplayer -- why are we talking about this?

I just wanted to drive home the concept of what a Minecraft server is and what it does. Now that we understand that, we can move on to a more relevant use case for a server -- a development server!

When we write code to modify Minecraft, we need to test to make sure the code works the way we expect when the game is running; that's what a development server allows us to do. We're connecting a server whose sole purpose is for testing our mod code. Since it's just for testing and development, the development server is run from your local machine.

There's no need for the development server once you're done developing your mod.

Let's pretend that we want to code a mod that we eventually play with friends; here's what that might look like:

• install an instance of Minecraft locally via the Fabric Launcher (we did this in the last step)

• set up our development environment for Java in VS Code

• spin up a local development server that has the Fabric API running on it

• set up a network-accessible server that has the Fabric API running on it

• develop our mod against the local Fabric development server

• once we're finished with the mod, it will exist in a .jar file

• we can test the finished mod in single-player by putting that .jar file into our /mods folder within our Fabric Launcher installation

• we decide the mod is ready for prime time, and copy the .jar file into the /mods folder on our network-accessible Fabric server

• our friends connect to the server and play our mod

We've already installed the local Minecraft instance with the Fabric Launcher. The next step is to set up our development environment in VS Code.

Let's get into it!

Development Environment

This guide will cover how to set up your development environment for VS Code. If you're using something else, you can find instructions on the Fabric wiki .

Your development environment needs to be on the computer where Minecraft is installed. This is because running the Java project in VS Code (or whatever you're using) opens the Minecraft client. The Minecraft client then interacts with the development server so that you can test the current state of your mod.

A note on WSL

If you're not a Windows user (or don't use WSL), you can skip ahead to the next section. If you're a Windows user that normally develops in WSL (like me), you'll need to use plain old Windows for your development environment -- not WSL.

I came to this conclusion after many hours of troubleshooting. I never found an official answer to validate my assumptions, but what I gather is this:

When you run your mod's Java project, VS Code opens up Minecraft and temporarily loads your mod into the game so you can test it. For VS Code to be able to do that, it needs to be able to "see" where Minecraft is installed on your computer.

Even though the Windows file system can be accessed via WSL, Microsoft doesn't recommend cross-communication between the two systems. They're meant to be used independently. Because of this, you'll get errors if you try and run the Java project from WSL (since Minecraft is installed on the Windows filesystem).

For this same reason, if you're running your (non-development) Fabric server locally, you'll want to run the server from within Windows and not WSL. This is because Windows can't "see" that there is a server running on WSL (unless you're running the server in a Docker container). The blog post on running a Minecraft server should provide some more context.

Setting up VS Code

In this section, we download an example mod template and make the necessary setup and configuration changes that allow us to run the mod from VS Code.

We start by following the 'Manual Steps' on the wiki

(I'm doing this from a Powershell prompt on Windows):

• go to the example mod git repo

• clone the repo

  • git clonehttps://github.com/FabricMC/fabric-example-mod.git

• change into the repo directory we just cloned

  • cd fabric-example-mod

• open the repo in VS Code

  • code . (there is a space between the word code and the period)

• delete the LICENSE and README.md files

Before we start developing, we need to make sure VS Code is set up for Java by installing the necessary extensions.

• Extension Pack for Java

• Gradle for Java

Go ahead and install the extensions. You may need to reload VS Code once they are finished installing for the changes to take effect.

You might see an error like the one below, but we'll fix that shortly:

The supplied phased action failed with an exception.
A problem occurred configuring root project 'fabric-example-mod'.
Failed to setup Minecraft, java.nio.file.ClosedFileSystemException: null

Before we address the error, let's modify the mappings in our gradle.properties file. Fabric helps us out by providing the values on this page.

The instructions say we also need to edit the maven_group and archives_base_name properties in our gradle.properties file. These settings are related to how a mod is packaged, versioned, and distributed. Since we're not publishing a mod, and we're only concerned with the fundamentals of how to make a mod in this guide, we're not going to concern ourselves with these settings for now.

In other words, just put something there. Here's what my file looks like:

# gradle.properties

# Done to increase the memory available to gradle.
org.gradle.jvmargs=-Xmx1G
org.gradle.parallel=true

# Fabric Properties
# check these on https://fabricmc.net/develop
minecraft_version=1.19.4
yarn_mappings=1.19.4+build.2
loader_version=0.14.19

# Mod Properties
mod_version = 1.0.0
maven_group = sharif.sharif.test.mod
archives_base_name = sharif-test-mod

# Dependencies
fabric_version=0.82.0+1.19.4

After saving the file, this message appears in the corner of VS Code. Press 'Yes':

We then see this message in the 'Problems' tab of the VS Code terminal area:

Now we can address the error.

If we look at Fabric's VS Code Setup documentation, it tells us that we need to run the vscode task.

./gradlew vscode

Once this command runs, the error should go away (close and re-open VS Code if it doesn't). The error was because we hadn't run the Gradle configuration that was specific to VS Code.

Now we should be able to run the following command without any issues:

./gradlew --stop # press enter
./gradlew # press enter

# the script runs for a bit, then gives the following message when finished
BUILD SUCCESSFUL in 12s
1 actionable task: 1 executed

If ./gradlew fails and your'e on Windows, you may just need to run ./gradlew.bat. If it fails and you're not on Windows, it's probably a problem with your JAVA_HOME environment variable.

Now we're ready to generate the Minecraft source code

./gradlew -- stop # just in case
./gradlew genSources # this will run for awhile

When we ran the genSources command, it downloaded the Minecraft source code onto our computer. We'll dig into that a bit more in a later section.

For now, we're finally able to run the project to see if it works. To do that, either press Ctrl + F5, or go to Run -> Run Without Debugging in VS Code.

You'll see some output in the terminal, and then Minecraft should open up.

Alright, so we've run the build, but how do we actually make a mod?!

Patience, young Glow Squid; we do that next.

Making Modifications

The moment we've all been waiting for!

Below, we add a new item to the game by following the tutorial provided by the Fabric wiki.

Adding an Item

This section is merely a more detailed version of the instructions for adding an item to the game.

• open our ExampleMod.java file

  • can press Ctrl + P and search for 'Example'

  • or navigate directly: /src/main/java/net/fabricmc/example/ExampleMod.java

Once we make the suggested changes to the file, it looks like this:

package net.fabricmc.example;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import net.fabricmc.api.ModInitializer;
import net.fabricmc.fabric.api.item.v1.FabricItemSettings;
import net.minecraft.item.Item;
import net.minecraft.registry.Registries;
import net.minecraft.registry.Registry;
import net.minecraft.util.Identifier;

public class ExampleMod implements ModInitializer {

    public static final Logger LOGGER = LoggerFactory.getLogger("modid");

    // an instance of our new item
    public static final Item CUSTOM_ITEM = new Item(new FabricItemSettings());

    @Override
    public void onInitialize() {

        LOGGER.info("Hello Fabric world!");
        Registry.register(Registries.ITEM, new Identifier("tutorial", "custom_item"), CUSTOM_ITEM);

    }
}

Now we can go to the 'Run' menu in VS Code and select 'Run without Debugging' again.

Once the game loads, we enter singleplayer mode and type the following command (type the / key to bring up the in-game chat):

/give @s tutorial:custom_item

You should see the item appear in your player's hand and inventory!

If you want to do more things with the new item, like add a new texture to the item, trigger sounds with the item, etc., go ahead and continue with the instructions on the Fabric wiki. For the scope of this blog post, we're only interested in the fact that we were able to successfully make a mod.

Getting your mod into a .jar file

Now that we've made a mod, we want to be able to use it in an actual instance of Minecraft, not just the sandboxed world that we get when we run the mod in VS Code.

All mods (including Fabric itself) are packaged in a .jar file. That means we need to get our mod into a .jar file.

To do that, we need to run the 'build' task:

• in VS Code, right-click on the build.gradle file

  • select 'Show Gradle Tasks'

  • run the 'build' task

• build is output to build/libs

• the file you want is the one without -sources in the filename

Now that our mod is nicely packaged into that .jar file, we're ready to put it in the /mods folder.

• if using the mod in single-player mode, this refers to the /mods folder within the installation directory of the Fabric installation we set up in the Fabric Launcher (Client) section above

  • if you installed the Mod Menu mod in that section, you can use the 'open mods folder' button that is available from the Mod Menu interface

• if using the mod on a server, this refers to the /mods directory on the server

  • the way this directory is accessed varies depending on how the server is set up

  • more info in Running a Minecraft Server

Once your .jar file is in its appropriate /mods folder, we're ready to test it out:

• create a new world with cheats enabled (so that we can use commands)

• run the /give command that we ran earlier when we were in our development environment

  • /give @s tutorial:custom_item

If you look at the Mod Menu interface after copying your .jar file to the /mods folder, you will have noticed that the mod still has all of the default tutorial values -- the name, thumbnail and descriptions are all generic.

A good way to get comfortable with the structure of a Fabric project is to try and change all of these values to something of your choosing, then re-install the mod with your new changes.

Other Ways to Modify Minecraft

Now that we've successfully created a Minecraft mod, I think it's important to understand how that fits into the bigger picture of the modding paradigm.

We can think of a modding framework like Fabric as having many layers. We make changes at whatever layer is appropriate for the mod in question. More complex mods require making changes at deeper layers.

To better understand these layers, take a look at the Introduction to Modding page in the wiki.

The recommended approach is to start at the outer layer (Native Minecraft APIs), and only move to the next layer if the current layer doesn't meet your needs.

For example, the item we created in the last section made use of the Registry, a Native Minecraft API.

The wiki does a better job than I can at explaining most of the other layers, but I do want to call attention to Mixins because it draws on some fundamental Java knowledge that you may not already be equipped with.

Mixins

If you looked at the wiki, you will have seen that Mixins are the innermost layer of the modding framework, and that Mixins allow the Minecraft source code to be changed.

I wanted to bring this up because it ties back into the genSources command we ran earlier when setting up our development environment in VS Code. The genSources command downloads the Minecraft source code onto our computer so that we can view it (see FAQ here).

Before Mixins were a thing, the source code needed to be modified directly. As you might imagine, this can lead to problems. Thus, Mixins were created as a layer of abstraction so that the source code could be modified more safely and predictably.

Since we don't need to modify the source code directly, reading the source code isn't as critical as it was before Mixins came along, but it's still good to know how everything works behind the scenes.

Bytecode

Before we get into viewing the bytecode, it's important to understand what bytecode is.

Bytecode is what allows Java to run on any platform, by way of the JVM (Java Virtual Machine); it's an intermediary step between a .java file and binary. Java bytecode is represented by a .class file.

For example, if we have a .java file on our Windows machine, and we want to share it with someone on MacOS, simply giving them the .java file won't work because it's a different operating system and therefore processes .java files differently.

This is where the JVM comes into play.

Windows, MacOS, and Linux all have their own OS-specific versions of the JVM. So if we go back to our scenario, but this time make use of the JVM, it looks something like this:

• we have a .java file on our Windows machine

• the JVM converts our .java file to a .class (bytecode) file

• we share the .class file with someone on MacOS

  • Java is installed on the Mac

• because Java is installed, the Mac can use the Mac version of the JVM to convert the .class file into binary code that MacOS understands

Again, you probably won't need to dig into bytecode, even if you do need to make modifications at the Mixin level, but you never know when it might come in handy.

By default, VS Code is not set up to view bytecode, but it can be enabled with an extension.

Closing thoughts

I'm aware that this blog post probably seemed like a whole lot of talking and setup, and not a lot of modding -- that was kind of the point. The setup and theory behind Minecraft mods were what I found to be the most challenging and elusive, and so that's what I've covered.

I may create more blog posts in the future that dive deeper into the details of the code, but the purpose of this blog post was to build a foundation and get you started.

Speaking of building a foundation, there was a lot of information that I couldn't logically fit into this blog post, so I've broken that info out into separate posts:

• Running a Minecraft Server (multiple approaches)

• Installing Java on Ubuntu (without breaking your shell)

These links are also included in the Resources section below.

Resources

• Running a Minecraft Server (multiple approaches)

• Installing Java on Ubuntu (without breaking your shell)

• Fabric example mod GitHub repo

• Fabric wiki

• VS Code Java extension pack

• JVM Bytecode Viewer extension

• How To Install Java with Apt on Ubuntu 22.04 (Digital Ocean blog)

• Suited Llamas Fabric Mod Tutorial

• Trouble Chute's 1.19+ Fabric Server Tutorial

TL;DR -- Different shells store environment variables in different locations. Set environment variables using your shell's recommended location.

There are plenty of resources out there for installing Java on Ubuntu, but I wanted to write one that covers the exact issue I ran into when installing Java with zsh.

This post focuses on the process for installing Java with zsh as compared to installing Java with bash. If you've no interest in zsh and just want to know how to bang it out with bash, I suggest reading this Digital Ocean post instead.

Update Package Manager

Before doing anything, it's a good idea to update our package manager:

sudo apt update
sudo apt upgrade

With that out of the way, the first thing I like to do is clear out any existing Java installations before making any other changes.

Check the existing Java installation

We can start by seeing what our existing Java setup looks like by issuing a command:

# WSL
java -version # press enter

# output
openjdk version "17.0.5" 2022-10-18
OpenJDK Runtime Environment (build 17.0.5+8-Ubuntu-2ubuntu122.04)
OpenJDK 64-Bit Server VM (build 17.0.5+8-Ubuntu-2ubuntu122.04, mixed mode, sharing)

Remove existing Java installation

You may or may not want to remove an existing Java installation. I prefer to start fresh, just in case.

In the previous step, I saw that I had the OpenJDK packages installed for Java, so I proceed with removing them.

Note: the comments add context (actions such as tab completions, saving files, re-opening the terminal)

# WSL
sudo apt remove --autoremove openjdk- # press tab repeteadly

# installed packages should be listed with tab completion

# after pressing tab, it autocompletes to:
sudo apt remove --autoremove openjdk-17-jre-headless:amd64 # press enter

# if more than one package appeared in the previous step, make sure to remove all of them

# source the shell (or close and re-open your terminal application)
source ~/.zshrc

# confirm Java is no longer installed
java -version # press enter

# output
zsh: command not found: java

Install desired Java version

Now we can install the appropriate version of Java (more specifically, the JDK). In my case, it's version 17.

So, we can just add 17 to openjdk- in the command below (replace 17 with whatever version you need to install):

sudo apt install openjdk-17 # press tab to see packages listed

# output
openjdk-17-dbg           openjdk-17-jdk           openjdk-17-jre-headless
openjdk-17-demo          openjdk-17-jdk-headless  openjdk-17-jre-zero
openjdk-17-doc           openjdk-17-jre           openjdk-17-source

# we want a headless JDK, we so end up with:
sudo apt install openjdk-17-jdk-headless

# source the shell or restart the terminal
source ~/.zshrc

# confirm Java installation
java -version # press enter

# output
openjdk version "17.0.7" 2023-04-18
OpenJDK Runtime Environment (build 17.0.7+7-Ubuntu-0ubuntu122.04.2)
OpenJDK 64-Bit Server VM (build 17.0.7+7-Ubuntu-0ubuntu122.04.2, mixed mode, sharing)

Set the $JAVA_HOME environment variable

We're not quite done yet, though. We need to set the $JAVA_HOME environment variable.

I'll first describe the way I initially tried it so that I can illustrate how much more straightforward the correct way is.

The wrong way

This is where I got thrown off by zsh. I didn't realize that zsh handles environment variables differently than bash.

I made the mistake of adding JAVA_HOME to my /etc/environment file (the bash way). The result went something like this:

# /etc/environment
JAVA_HOME="/path/to/java/home"

# save and exit /etc/environment

# source the file
source /etc/environment

echo $JAVA_HOME # outputs /path/to/java/home, yay!

# close terminal 
# open new terminal
echo $JAVA_HOME # no output, uhh.. wtf?

At this point, I thought I would get clever and source /etc/environment within my .zshrc

It almost worked:

# .zshrc
source /etc/environment

# save and exit .zshrc
# close and re-open terminal

echo $JAVA_HOME # success!

So far so good, but when I try to open a project with VS Code from the terminal...

code . # presss enter

zsh: command not found: code

GRRR!

I decide to get clever once again, and add my entire $PATH variable to my /etc/environment file:

# /etc/environment
PATH="/home/this/path:/home/that/path:/another/path:/vs/code/path"
JAVA_HOME="/java/home/path"

This solution actually worked, but it felt so hacky. I just knew there had to be a better way...

The right way

Before we jump in, note that the configuration has been reset from the previous section, (titledThe wrong way):

• source /etc/environment removed from .zshrc

• JAVA_HOME variable deleted from /etc/environment

• PATH deleted from /etc/environment

• the terminal was session closed and re-opened

Now we can go about setting JAVA_HOME properly:

echo $JAVA_HOME # press enter
# no output 

# find the Java install path
sudo update-alternatives --config java # press enter

# output
There is only one alternative in link group java (providing /usr/bin/java): 
/usr/lib/jvm/java-17-openjdk-amd64/bin/java
Nothing to configure.

From here we can copy the path that was output: /usr/lib/jvm/java-17-openjdk-amd64/bin/java

The place you put that path will depend on a few factors:

• which shell you're using

• whether this Java configuration is being set at the profile level or the system level

Luckily, what we put in the file is the same, regardless of scope or shell:

# at the bottom of the appropriate file from the table above
JAVA_HOME="/usr/lib/jvm/java-17-openjdk-amd64/bin/java"

Save and exit the file, then close and re-open the terminal.

Now we can see that JAVA_HOME is set if we echo it out:

echo $JAVA_HOME

# output
/usr/lib/jvm/java-17-openjdk-amd64/bin/java

That's it! Your Java configuration is ready to go, and your shell commands remain intact.

As you might imagine, there are a host of security concerns that come along with this requirement. I started looking for solutions and came across the idea of a chroot.

NOTE: I will be learning as I write this. This post is not intended to be a tutorial, but more a log of my working sessions and debugging efforts with this topic -- recording my thought process and what I learned along the way in a very informal matter.

What is chroot?

The idea is to create an isolated environment for a user (or users) so that they are limited in what they can see and do.

This is similar to the idea of a Docker container, but not quite the same.

A chroot lives inside an existing Linux install. You can almost think of it as an operating system within an operating system. Even if there are technically other users outside the chroot, the users inside the chroot are not aware of their existence. This is why a chroot is sometimes referred to as a "chroot jail".

The root user within a chroot is a sort of pseudo root. They do have root privileges within their sectioned-off operating system (the chroot jail), but those root privileges are being provisioned by the true root user, the one who set up the chroot.

Chroot creation -- first pass

I'm currently using a Raspberry Pi as my development server for this project, so that is where I'll be implementing the chroot.

I start by following this guide. The instructions are for Ubuntu (which is not what the Raspberry Pi is running), but the instructions here are more clear to me than others I've found so I'm going to try and make it work.

I install the schroot and debootstrap packages as instructed, create my /var/chroot directory, then open the /etc/schroot/schroot.conf file. I know I probably need to make some changes, but for now, I just leave it the way they suggest:

# /etc/schroot/schroot.conf

[lucid]
description=Ubuntu Lucid
location=/var/chroot
priority=3
users=your_username
groups=sbuild
root-groups=root

Here's the command they give to install the operating system within the chroot:

sudo debootstrap --variant=buildd --arch i386 lucid /var/chroot/ http://mirror.url.com/ubuntu/

The instructions say I need to replace mirror.url.com with a valid mirror, so I find a Raspbian mirror online and give it a go:

sudo debootstrap --variant=buildd --arch i386 lucid /var/chroot/ http://raspbian.mirrors.lucidnetworks.net/raspbian/ # press enter

# output -- error
E: Failed getting release file http://raspbian.mirrors.lucidnetworks.net/raspbian/dists/lucid/Release

At first, I thought that I just needed to find a Raspbian mirror that had /lucid under the /dists directory. I wasn't finding one. Then I thought that changing lucid to jammy would work, because I thought I saw jammy somewhere...

So I tried it

sudo debootstrap --variant=buildd --arch i386 jammy /var/chroot/ https://mirror.vcu.edu/pub/gnu_linux/ubuntu/ # press enter

# output -- error
E: No such script: /usr/share/debootstrap/scripts/jammy

Even though it didn't work, at least it was a new error!

I decided to look at the contents of the mentioned /scripts/ directory and see what was listed:

ls -la /usr/share/debootstrap/scripts # press enter

# output
drwxr-xr-x 2 root root 4096 May 16 06:40 .
drwxr-xr-x 3 root root 4096 May 16 06:40 ..
-rw-r--r-- 1 root root 6044 Jul 13  2022 aequorea
-rw-r--r-- 1 root root 6639 Jul 13  2022 amber
lrwxrwxrwx 1 root root    5 Jul 28  2022 artful -> gutsy
lrwxrwxrwx 1 root root    3 Jul 28  2022 ascii -> sid
lrwxrwxrwx 1 root root    8 Jul 28  2022 bartholomea -> aequorea
lrwxrwxrwx 1 root root    3 Jul 28  2022 beowulf -> sid
lrwxrwxrwx 1 root root    5 Jul 28  2022 bionic -> gutsy
lrwxrwxrwx 1 root root    3 Jul 28  2022 bookworm -> sid
-rw-r--r-- 1 root root 5404 Jul 13  2022 breezy
lrwxrwxrwx 1 root root    3 Jul 28  2022 bullseye -> sid
# ... <omitted output> ...

Although lucid is listed here (not shown in the output), I stopped when I saw bullseye because I know that is what my Raspberry Pi is running.

I also did not realize that the different names (lucid, bullseye, etc.) corresponded to different versions of Debian. After looking into it a bit, I learned that lucid is quite old (Ubuntu 10.04), while bullseye is much newer (18.04 +).

With that, I change my configuration:

# /etc/schroot/schroot.conf

[bullseye]
description=Raspbian Bullseye
location=/var/chroot
priority=3
users=sharif
groups=sbuild
root-groups=root

Changing lucid to bullseye is easy enough, but the mirror is a bit more involved. First, I need to find a list of mirrors, then find a mirror that is close to my geographical location.

I find the United States mirror and view its contents in the browser. From the error message I received previously, I know that the command looks for the version inside the dists/ directory from the provided mirror URL.

I navigate to the dists/ directory within the browser, and see that bullseye/ is listed.

Bingo!

So I go ahead and update my command with the correct version and mirror:

sudo debootstrap --variant=buildd --arch i386 bullseye /var/chroot/ http://ftp.us.debian.org/debian

It runs for a while and gets further than any of the previous attempts, but there's an issue at the very end:

W: Failure trying to run: chroot "/var/chroot" /bin/true
W: See /var/chroot/debootstrap/debootstrap.log for details

What happened here is that the bullseye version of Linux installed successfully within my /var/chroot/ directory, but the system failed to open a shell within the chroot environment.

I follow the output instructions and view the contents of debootstrap.log. At the end of the log is a more detailed error:

chroot: failed to run command ‘/bin/true’: Exec format error

I Google this error, and the suggested fix is to:

• install the qemu-user-static package (required for user emulation)

• sudo cp /usr/bin/qemu-arm-static /path/to/mount/usr/bin

  • needed to adjust the path on the right side for my use case:

  • sudo cp /usr/bin/qemu-arm-static /var/chroot/

• systemctl restart systemd-binfmt.service

Now I can try to launch the shell again:

sudo chroot /var/chroot # press enter

# output
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
I have no name!@raspberrypi:/#

I'm now logged in, but there's no user associated with the session:

whoami # press enter

# output
whoami: cannot find name for user ID 0: No such file or directory

Digging deeper

Going through the above guide did not get me where I wanted to be. It was a good start, but I wasn't sure how to add a user in the chroot, or why it was launched without a user in the first place.

The guide also referenced some mount commands that I wasn't familiar with.

The end of the guide also mentioned using schroot instead of chroot, but I felt like I could use a bit more detail.

The last thing that spooked me a bit was that it seemed to imply that doing things incorrectly could result in the accidental erasure of data. That's not what I want.

So I decided to poke around a bit for other resources so that I could broaden my understanding. Some of the references in the initial guide seemed to assume I was more familiar with Linux than I am, so maybe I can find some more beginner-friendly material.

I found this post, which seems to make things a bit more simple, so I follow along with its example.

Making another chroot

Following the Linode guide linked in the last section, I proceed to make a new chroot:

mkdir ~/chroot-jail # press enter 

sudo debootstrap bullseye ~/chroot-jail # press enter and wait for install

sudo chroot ~/chroot-jail /bin/bash

This time when the chroot launches, I am the root user. I also didn't need to mess with any mirrors or config files. So far, this is much better.

But I still need to learn how to do things in the chroot. I need to explore what I'm able to do, and what I'm not able to do. I also need to figure out what the users of my app should and shouldn't be able to do.

Lots of work ahead.

Let's keep following the guide; it says to install sudo, but I seem to need to do some work before I can get there because I get this error when I run apt install sudo:

E: Can not write log (Is /dev/pts mounted?) - posix_openpt (19: No such device)

Scrolling down in that section of the guide, I see that it instructs to mount the /dev/ directory inside the chroot, along with some other directories:

exit # exit the chroot

# back in main operating system
sudo mount --bind /proc ~/chroot-jail/proc/
sudo mount --bind /sys ~/chroot-jail/sys/
sudo mount --bind /dev ~/chroot-jail/dev/

# back into the chroot
sudo chroot ~/chroot-jail /bin/bash
apt install sudo # it works!

This is better so far, but now I'm starting to see the issue behind the mount points. In this configuration, a user could alter or delete a file in the /proc, /sys or /dev directories, and it would impact the main OS.

I can see that the next section in the guide mentions using schroot instead of chroot to limit the user's access.

schroot

I want to prevent the chroot user from having root access, even inside the chroot, as we have just seen that it has the potential to be dangerous.

Following the guide:

exit # exit the chroot

which schroot # make sure schroot is installed
/usr/bin/schroot # schroot is installed

sudo vim /etc/schroot/schroot.conf # edit the schroot.config file

[bullseye]
description=Raspbian Bullseye
location=/home/sharif/chroot-jail
priority=3
users=sharif
groups=sbuild
root-groups=root

# save the file and exit

schroot -c bullseye # run the schroot

# output

W: line 87 [bullseye] priority: Could not parse value ‘3’
W: line 86 [bullseye]: Obsolete key ‘location’ used
I: This option has been removed, and no longer has any effect
W: Failed to change to directory ‘/home/sharif’: No such file or directory
I: The directory does not exist inside the chroot.  Use the --directory option to run the command in a di
fferent directory.
W: Falling back to directory ‘/’
W: Shell ‘/usr/bin/zsh’ not available: /usr/bin/zsh: Failed to stat file: No such file or directory
W: Falling back to shell ‘/bin/bash’

Going through the output line by line, I can start to see what's happening:

• W: line 87 [bullseye] priority: Could not parse value ‘3’

  • the priority line was leftover from the first guide I followed. I'll remove it

• W: line 86 [bullseye]: Obsolete key ‘location’ used

• I: This option has been removed, and no longer has any effect

  • looks like another issue from the previous config file. I see in the new guide that the key has been updated to directory instead of location, so I make that change

• W: Failed to change to directory ‘/home/sharif’: No such file or directory

• I: The directory does not exist inside the chroot. Use the --directory option to run the command in a different directory.

  • it looks like I misunderstood what I was specifying in the config file. I thought I was pointing to the directory that the chroot was under, but it instead wants the directory to load inside the chroot

  • I assume I will need to first create a user inside the chroot, then point to that user's directory in the config file

• W: Falling back to directory ‘/’

  • since there was no valid user for the chroot to load, we default to the root user directory

• W: Shell ‘/usr/bin/zsh’ not available: /usr/bin/zsh: Failed to stat file: No such file or directory

• W: Falling back to shell ‘/bin/bash’

  • I believe what happened here is that it tried to load zsh as my shell from the main operating system, because that is my user's default shell

  • This error should go away once I create a new user because their default shell will be bash and not zsh

So the first thing I need to do is create a user within the chroot. I remember now that I had to follow the guide's instructions out of order, and that there were steps on creating a user in the guide:

adduser chroot-test sudo
-bash: adduser: command not found

man useradd
-bash: man: command not found

$ which useradd
$ man useradd
-bash: man: command not found

Looks like I'm not able to add users within the chroot. I'll try installing the package(s) it calls out:

$ apt-get install man
E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?
$ pwd
/
$ whoami
whoami: cannot find name for user ID 1000

Ah, that's right. I'm still in the chroot that was launched with schroot, not the initial one I launched to install sudo. So I'll switch back to that:

exit #exit schroot

sudo chroot ~/chroot-jail /bin/bash # enter elevated chroot

useradd -m chroot-user # add user with home directory

ls /home # test to see if home directory was created
# output
chroot-user

usermod -aG sudo chroot-user # add user to sudo group

groups chroot-user # make sure sudo is listed
#output
chroot-user : chroot-user sudo

Now I need to exit the chroot, go back to the schroot configuration file, and see if it works with my new user:

exit # exit chroot
sudo vim /etc/schroot/schroot.conf # edit the schroot.config file

[bullseye]
description=Raspbian Bullseye
directory=/home/sharif/chroot-jail
users=chroot-user
groups=sbuild
root-groups=root

# save and exit the file

schroot -c bullseye

E: Access not authorised
I: You do not have permission to access the schroot service
I: This failure will be reported.

sudo schroot -c bullseye

W: Failed to change to directory ‘/home/sharif’: No such file or directory
I: The directory does not exist inside the chroot.  Use the --directory option to run the command in a different directory.
W: Falling back to directory ‘/root’

exit # exit the chroot

sudo vim /etc/schroot/schroot.conf # edit the schroot.config file

[bullseye]
description=Raspbian Bullseye
directory=/home/chroot-user
users=chroot-user
groups=sbuild
root-groups=root

# save and exit the file

sudo schroot -c bullseye
E: Failed to change to directory ‘/home/chroot-user’: No such file or directory

Seems to be some sort of circular logic here. I look back at the guide and see that it specifically says the user we are using in the chroot must match the user we are using to access the chroot.

I read this as: I need to also create a chroot-user in the actual operating system before I can use it in the chroot.

But, I'm tired, and sort of did the opposite by adding a sharif user to the chroot. Let's look at the effects of this mistake before correcting it:

sudo chroot ~/chroot-jail /bin/bash # enter chroot


useradd -m sharif

ls /home
chroot-user  sharif

groups sharif
sharif : sharif

sudo usermod -aG sudo sharif

exit # exit chroot

sudo vim /etc/schroot/schroot.conf 

[bullseye]
description=Raspbian Bullseye
directory=/home/sharif/chroot-jail
users=sharif
groups=sbuild
root-groups=root

# save and exit the file

schroot -c bullseye # enter chroot

W: Shell ‘/usr/bin/zsh’ not available: /usr/bin/zsh: Failed to stat file: No such file or directory
W: Falling back to shell ‘/bin/bash’
chroot-user@raspberrypi:~$ pwd
/home/sharif
chroot-user@raspberrypi:~$ whoami
chroot-user
chroot-user@raspberrypi:~$ cd ..
chroot-user@raspberrypi:/home$ ls
chroot-user  sharif
chroot-user@raspberrypi:/home$ cd sharif/
chroot-user@raspberrypi:~$ ls -la
total 20
drwxr-xr-x 2 sharif sharif 4096 May 17 11:59 .
drwxr-xr-x 4 root   root   4096 May 17 11:59 ..
-rw-r--r-- 1 sharif sharif  220 Mar 27  2022 .bash_logout
-rw-r--r-- 1 sharif sharif 3526 Mar 27  2022 .bashrc
-rw-r--r-- 1 sharif sharif  807 Mar 27  2022 .profile
chroot-user@raspberrypi:~$ pwd
/home/sharif
chroot-user@raspberrypi:~$ cd ../chroot-user/
chroot-user@raspberrypi:/home/chroot-user$ ls -la
total 20
drwxr-xr-x 2 chroot-user chroot-user 4096 May 17 11:41 .
drwxr-xr-x 4 root        root        4096 May 17 11:59 ..
-rw-r--r-- 1 chroot-user chroot-user  220 Mar 27  2022 .bash_logout
-rw-r--r-- 1 chroot-user chroot-user 3526 Mar 27  2022 .bashrc
-rw-r--r-- 1 chroot-user chroot-user  807 Mar 27  2022 .profile
chroot-user@raspberrypi:/home/chroot-user$ exit

So the problem is that I am logged into the chroot as the correct user, chroot-user, but there are some issues

• the chroot starts in the /home/sharif directory

• the sharif user should not be on the system (I want a dedicated chroot for each user)

The first logical step is to remove the sharif user from the chroot:

schroot -c bullseye
cd /home/chroot-user

sudo userdel -r sharif

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

[sudo] password for chroot-user:

Oops, I never set a password when I created chroot-user. Let's see if I can fix that from within the chroot:

passwd chroot-user
Changing password for chroot-user.
Current password:

Hm, I'm prompted for a password when trying to change the password. That won't do. Let me try in the original chroot:

exit

sudo chroot ~/chroot-jail /bin/bash

ls /home
chroot-user  sharif

userdel -r sharif
userdel: sharif mail spool (/var/mail/sharif) not found

# userdel gives an error, but the deletion still worked

ls /home
chroot-user

# try setting a password for chroot-user now as root
passwd chroot-user
New password:
Retype new password:
passwd: password updated successfully

cat /etc/passwd # confirm that user sharif is not listed

Now I can do what I should have done originally -- create chroot-user on the base OS

exit # exit chroot

sudo adduser chroot-user # accept all default prompts
sudo adduser chroot-user sudo

Now I need to adjust the schroot config again to make sure it points to the right user directory:

sudo vim /etc/schroot/schroot.conf

[bullseye]
description=Raspbian Bullseye
directory=/home/sharif/chroot-jail
users=chroot-user # the only line that was changed
groups=sbuild
root-groups=root

# save and exit the file

schroot -c bullseye

# output 
E: Access not authorised
I: You do not have permission to access the schroot service.
I: This failure will be reported.

sudo schroot -c bullseye
W: Failed to change to directory ‘/home/sharif’: No such file or directory
I: The directory does not exist inside the chroot.  Use the --directory option to run the command in a different directory.
W: Falling back to directory ‘/root’

I think I finally understand the issue (at least partially).

It's trying to start the chroot from the base OS in the directory /home/sharif/chroot-jail, which does exist -- but that path (up until /chroot-jail also needs to be mirrored in the chroot OS, and since I removed thesharif user from the chroot, it no longer exists.

I believe the directory key will force the directory to be created on the base OS if it isn't already, but it can't create a folder in the chroot OS.

Time to try it out

sudo vim /etc/schroot/schroot.conf

[bullseye]
description=Raspbian Bullseye
directory=/home/chroot-user/chroot-jail # changed this line
users=chroot-user 
groups=sbuild
root-groups=root

# save and exit the file

schroot -c bullseye

E: Access not authorised
I: You do not have permission to access the schroot service.
I: This failure will be reported.

sudo schroot -c bullseye

E: Failed to change to directory ‘/home/chroot-user/chroot-jail’: No such file or directory

Okay, so I think the issue is that I set up the chroot-jail folder under my sharif user, and I now need to move it to chroot-user for the above to work:

sudo mv /home/sharif/chroot-jail /home/chroot-user/

ls ~/ # confirm chroot-user is no longer in my user's home directory

ls /home/chroot-user
chroot-jail

schroot -c bullseye
E: Access not authorised
I: You do not have permission to access the schroot service.
I: This failure will be reported.

sudo schroot -c bullseye
W: Failed to change to directory ‘/home/sharif’: No such file or directory
I: The directory does not exist inside the chroot.  Use the --directory option to run the command in a different directory.

And now I am back to being confused. I am not sure why the chroot is trying to access /home/sharif because it is not referenced in the configuration file.

I source my shell, then get a more fitting error message:

source ~/.zshrc

sudo schroot -c bullseye

E: Failed to change to directory ‘/home/chroot-user/chroot-jail ’: No such file or directory

ls /home/chroot-user
chroot-jail

The folder looks like it's there. so maybe it's a permissions issue?

 ls -la /home/chroot-user

drwxr-xr-x  3 chroot-user chroot-user 4096 May 17 09:55 .
drwxr-xr-x  5 root        root        4096 May 17 09:41 ..
-rw-r--r--  1 chroot-user chroot-user  220 May 17 09:41 .bash_logout
-rw-r--r--  1 chroot-user chroot-user 3523 May 17 09:41 .bashrc
drwxr-xr-x 17 root        root        4096 May 17 05:48 chroot-jail

Maybe it's because the chroot-jail folder is owned by root?

sudo chown chroot-user:chroot-user /home/chroot-user/chroot-jail

ls -la /home/chroot-user

drwxr-xr-x  3 chroot-user chroot-user 4096 May 17 09:55 .
drwxr-xr-x  5 root        root        4096 May 17 09:41 ..
-rw-r--r--  1 chroot-user chroot-user  220 May 17 09:41 .bash_logout
-rw-r--r--  1 chroot-user chroot-user 3523 May 17 09:41 .bashrc
drwxr-xr-x 17 chroot-user chroot-user 4096 May 17 05:48 chroot-jail
-rw-r--r--  1 chroot-user chroot-user 1670 May 17 09:41 .mkshrc
-rw-r--r--  1 chroot-user chroot-user  807 May 17 09:41 .profile

schroot -c bullseye
E: Access not authorised
I: You do not have permission to access the schroot service.
I: This failure will be reported.

sudo schroot -c bullseye
E: Failed to change to directory ‘/home/chroot-user/chroot-jail ’: No such file or directory

sudo vim /etc/schroot/schroot.conf 

# change directory to /home/chroot-user (take off '/chroot-jail')

schroot -c bullseye

E: Access not authorised
I: You do not have permission to access the schroot service.
I: This failure will be reported.

sudo schroot -c bullseye

W: Failed to change to directory ‘/home/sharif’: No such file or directory
I: The directory does not exist inside the chroot.  Use the --directory option to run the command in a different directory.
W: Failed to change to directory ‘/root’: No such file or directory
I: The directory does not exist inside the chroot.  Use the --directory option to run the command in a different directory.
W: Falling back to directory ‘/’
W: Shell ‘/bin/bash’ not available: /bin/bash: Failed to stat file: No such file or directory
W: Shell ‘/bin/sh’ not available: /bin/sh: Failed to stat file: No such file or directory
W: Falling back to shell ‘/bin/sh’
E: Failed to execute “/bin/sh”: No such file or directory

Now I get even more errors! My understanding of the configuration needs a little work. One thing that I can say is that it seems related to file permissions between the base OS and the chroot.

I decide to look up the documentation for schroot, and it looks like I might need to specify my chroot type as directory :

# /etc/schroot/schroot.conf
[bullseye]
description=Raspbian Bullseye
type=directory # added this line
directory=/home/chroot-user/chroot-jail # changed this line
users=chroot-user
groups=sbuild
root-groups=root
# save the file

schroot -c bullseye
E: Access not authorised
I: You do not have permission to access the schroot service.
I: This failure will be reported.

sudo schroot -c bullseye

I know that I should be able to run the schroot -c bullseye command without elevated permissions. When I get an error, I know that there is something amiss with the configuration. However, when I run it with sudo, I typically get some more information about why the command failed.

This time, when I run it with sudo, I get no errors, but I still don't get the desired behavior. For one, it puts me in the /home/sharif directory on the base OS, not the chroot. Even worse, when I run a whoami, I get back root.

I need to understand why this is happening. I think I will try un-commenting one of the more basic-looking configs in the schroot.conf file and see what happens:

[sid]
description=Debian sid (unstable)
directory=/srv/chroot/sid
users=rleigh # changing this to 'chroot-user'
groups=sbuild
root-groups=root
aliases=unstable,default
# save the file

schroot -c sid

E: Access not authorised
I: You do not have permission to access the schroot service.
I: This failure will be reported.

sudo schroot -c sid
E: Failed to change to directory ‘/srv/chroot/sid’: No such file or directory

Running the command with sudo gives an error that the directory doesn't exist, so I create it and try again:

sudo mkdir -p /srv/chroot/sid

sudo schroot -c sid

W: Failed to change to directory ‘/home/sharif’: No such file or directory
I: The directory does not exist inside the chroot.  Use the --directory option to run the command in a different directory.
W: Failed to change to directory ‘/root’: No such file or directory
I: The directory does not exist inside the chroot.  Use the --directory option to run the command in a different directory.
W: Falling back to directory ‘/’
W: Shell ‘/bin/bash’ not available: /bin/bash: Failed to stat file: No such file or directory
W: Shell ‘/bin/sh’ not available: /bin/sh: Failed to stat file: No such file or directory
W: Falling back to shell ‘/bin/sh’
E: Failed to execute “/bin/sh”: No such file or directory

So, this is strange... even though there is no reference to /home/sharif in the config file, it is looking for that directory in the chroot. Did I maybe mess something up with the chroot-user ?

I decide to look back at the guide I was originally following, and start fresh. Maybe I will notice something that I didn't before, now that I have a bit more experience with it.

I start by deleting the chroot-jail folder that I tried to assign to the chroot-user, delete the user, recreate the chroot-jail folder, and reinstall the chroot OS there:

sudo rm -rf /home/chroot-user/chroot-jail

sudo userdel -r chroot-user

mkdir ~/chroot-jail

sudo debootstrap bullseye ~/chroot-jail

The install runs for a while, but ends with an error when trying to run the chroot:

W: Failure trying to run: chroot "/home/sharif/chroot-jail" dpkg --force-overwrite --force-confold --skip-same-version -
-install /var/cache/apt/archives/libapparmor1_2.13.6-10_armhf.deb /var/cache/apt/archives/libargon2-1_0~20171227-0.2_arm
hf.deb /var/cache/apt/archives/libcryptsetup12_2%3a2.3.7-1+deb11u1_armhf.deb /var/cache/apt/archives/libip4tc2_1.8.7-1_a
rmhf.deb /var/cache/apt/archives/libjson-c5_0.15-2_armhf.deb /var/cache/apt/archives/libkmod2_28-1_armhf.deb /var/cache/
apt/archives/libcap2_1%3a2.44-1_armhf.deb /var/cache/apt/archives/dmsetup_2%3a1.02.175-2.1_armhf.deb /var/cache/apt/arch
ives/libdevmapper1.02.1_2%3a1.02.175-2.1_armhf.deb /var/cache/apt/archives/systemd_247.3-7+deb11u2_armhf.deb

W: See /home/sharif/chroot-jail/debootstrap/debootstrap.log for details (possibly the package systemd is at fault)

The first message is a bit tough to unpack, so I go to the second, which gives me a file path for a log that I can view:

cat /home/sharif/chroot-jail/debootstrap/debootstrap.log # press enter

# last part of output:
dpkg: error processing package systemd (--install):
 installed systemd package post-installation script subprocess returned error exit status 73

Processing triggers for libc-bin (2.31-13+deb11u6) ...
Errors were encountered while processing:
 systemd

The dpkg error seems more specific, so I start by googling that.

I end up on this post, and give the accepted answer a try:

stat / /dev /var # press enter

It looks like root is the owner of all those directories, so I believe my problem is a different one. The post above gives credit to another post, which has another answer that includes removing /var/lib/dpkg/info/systemd*. Somebody in the comments said this fixed the problem, but now they have warnings about missing files when they use apt-get to install something. I'm not sure if I like this solution.

I decide to go back to my Google search results. Another post suggests running sudo apt update:

sudo apt update

Now I should be able to try and run the chroot again since the install worked; the problem is with launching the chroot. Time to see if the update fixed it:

sudo chroot ~/chroot-jail /bin/bash

This works, but now the problem is that I'm using chroot, which gives full access to the chroot, when I need to be using schroot for limited access.

I continue following the instructions in the guide:

# in the chroot
adduser sharif# go through the prompts to set a password and info
adduser sharif sudo

# exit chroot
exit

# mount per instructions
sudo mount --bind /proc ~/chroot-jail/proc/
sudo mount --bind /sys ~/chroot-jail/sys/
sudo mount --bind /dev ~/chroot-jail/dev/

# /etc/schroot/schroot.conf
[bullseye]
description=Raspbian Bullseye
directory=/home/sharif/chroot-jail
users=sharif
groups=sbuild
root-groups=root
aliases=bullseye
# save the file

# run schroot
schroot -c bullseye

It works! The issue was that the user you are launching the command as in the base operating system must match the user in the chroot. I was only halfway there before since I had a matching user in the chroot, but I was not executing the command as that user from the base OS.

Now, the issue is that I don't think the sharif user in the chroot should have root privileges, but my sharif user in the base OS does have root privileges. I think I need to do this again, but with a non-elevated user on both sides:

# remove the chroot-jail folder for my sharif user
sudo rm -rf ~/chroot-jail

# output gives a bunch of 'Permission denied' errors

Ah, that's right. I need to follow the instructions for removing a chroot.

sudo umount ~/chroot-jail/dev
sudo umount ~/chroot-jail/sys
sudo umount ~/chroot-jail/proc

# output
git_prompt_info:3: permission denied: /dev/null
# end output
sudo rm -R ~/chroot-jail

git_prompt_info:3: permission denied: /dev/null

It looks like the removal worked, but I'm not sure what the error is for, so back to Google..

Looks like rebooting the system will do the trick:

sudo reboot

Now I need to create the non-root user in the base OS, install the chroot jail under their user this time, then add the user in the chroot:

sudo adduser schrooty # press enter

# go through propts to set password, name etc.

# switch users
su - schrooty # press enter
# enter password

# follow chroot instructions
mkdir ~/chroot-jail

And it is at this point that I realize that I was incorrect in my thought process. Following the instructions for setting up the chroot, sudo is required. That means that I either need to:

• run the commands as my sharif user in the base OS, and point to /home/schrooty/root-jail when I install and run chroot

• create another user that I add to sudo, which will be used for managing the chroot (I don't want to manage it with my normal user account that I use for everything else)

It's also worth calling out that I'm not sure if the chroot user needs to own the chroot-jail folder, or if it needs to be owned by root. I assume it needs to be owned by the chroot user, as the instructions do not say to run the mkdir command with sudo

It may also be the case that the folder needs to be owned by whatever user is spawning the chroot.

# switch back to sharif user
su - sharif # press enter
# enter password

# follow instructions, but pointing to schrooty
sudo debootstrap bullseye /home/schrooty/chroot-jail

# output
W: Failure trying to run: chroot "/home/schrooty/chroot-jail" dpkg --force-overwrite --force-confold --skip-same-version --install /var/cache/apt/archives/libapparmor1_2.13.6-10_armhf.deb /var/cache/apt/archives/libargon2-1_0~20171227-0.2_armhf.deb /var/cache/apt/archives/libcryptsetup12_2%3a2.3.7-1+deb11u1_armhf.deb /var/cache/apt/archives/libip4tc2_1.8.7-1_armhf.deb /var/cache/apt/archives/libjson-c5_0.15-2_armhf.deb /var/cache/apt/archives/libkmod2_28-1_armhf.deb /var/cache/apt/archives/libcap2_1%3a2.44-1_armhf.deb /var/cache/apt/archives/dmsetup_2%3a1.02.175-2.1_armhf.deb /var/cache/apt/archives/libdevmapper1.02.1_2%3a1.02.175-2.1_armhf.deb /var/cache/apt/archives/systemd_247.3-7+deb11u2_armhf.deb

W: See /home/schrooty/chroot-jail/debootstrap/debootstrap.log for details (possibly the package systemd is at fault)

It's the same error I got last time. I guess running the update didn't do anything. I decide to ignore the warning and try to run the chroot:

sudo chroot /home/schrooty/chroot-jail /bin/bash # press enter

# successfully launched chroot
# now in chroot
adduser schrooty # press enter
# go through prompts to create user and set password
# exit chroot
exit

# back in base OS (as user sharif)
sudo mount --bind /proc /home/schrooty/chroot-jail/proc/
sudo mount --bind /sys /home/schrooty/chroot-jail/sys/
sudo mount --bind /dev /home/schrooty/chroot-jail/dev/

# configure schroot
sudo vim /etc/schroot/schroot.conf

[bullseye]
description=Raspbian Bullseye
directory=/home/schrooty/chroot-jail
users=schrooty
groups=sbuild
root-groups=root
aliases=bullseye
# save and exit the file

# attempt to launch schroot
schroot -c bullseye
W: line 91 [bullseye] aliases: Alias ‘bullseye’ already associated with ‘unknown’ chroot
E: Access not authorised
I: You do not have permission to access the schroot service.
I: This failure will be reported.

# is the problem with the alias?
sudo vim /etc/schroot/schroot.conf
# remove alias and save file

# nope, didn't fix it
schroot -c bullseye
E: Access not authorised
I: You do not have permission to access the schroot service.
I: This failure will be reported.

# try with sudo
sudo schroot -c bullseye
W: Failed to change to directory ‘/home/sharif’: No such file or directory
I: The directory does not exist inside the chroot.  Use the --directory option to run the command in a different directory.
W: Falling back to directory ‘/root’

# maybe I need to run this as schrooty?
# exit chroot (since it launched as root -- not what I want)
exit
su - schrooty
schroot -c bullseye # success!

It works this time, but there are some issues:

• the prompt shows up as I have no name!@raspberrypi

• running groups yields: groups: cannot find name for group ID 1003

• running whoami yields: whoami: cannot find name for user ID 1002

However, the chroot did launch in the correct directory: /home/schrooty. The problem seems to be that it's not launching the session as schrooty.

After looking online, I learn that is not the reason for this behavior. It is because files like /etc/passwd and /etc/shadow are not present in the chroot, so they need to be mounted like the other directories from the config instructions.

I will keep this post as reference, as the answer was given by the author of the schrooty package

# exit chroot
exit

# switch back to sharif user so I can use sudo
su - sharif 
# enter password

# mount files
sudo mount --bind /etc/passwd /home/schrooty/chroot-jail/etc/passwd
sudo mount --bind /etc/shadow /home/schrooty/chroot-jail/etc/shadow

# switch back to schrooty
su - schrooty
# enter password

# launch chroot
schroot -c bullseye

# prompt now has username -- schrooty@raspberrypi:~ $
whoami
# output
schrooty

groups
#output
groups: cannot find name for group ID 1003

Much better! Except I now need to copy the file for groups over to the chroot:

# exit chroot
exit

# switch users
su - sharif

# mount file
sudo mount --bind /etc/group /home/schrooty/chroot-jail/etc/group

# switch users
su - schrooty

# luanch chroot
schroot -c bullseye

pwd # outputs /home/schrooty
whoami # outputs schrooty
groups # outputs schrooty

After alllll of this, I decided to look into how secure chroot (even with schroot) actually is...

And it's not good. It looks relatively easy to exploit.

It looks like a Docker container is going to give me what I'm after, but I still think this was a good exercise.

So, stay tuned for the Docker version of this one I guess 😅

But now I'm done with that blog post, and root is still available for login with a password (I didn't use my real domain name in that blog post, calm down).

Naturally, I'd like to lock it back down to the status it was in before I wrote that blog post. I figured I'd document the process and share my findings along the way.

Prework

Before disabling root login on the Vultr instance, I need to make sure that I'm able to ssh in as a non-root user.

This is because some ssh configuration has to be done as the root user on the Vultr instance. Once that configuration is done, then we can be done with root.

Vultr documentation

Referencing Vultr's documentation for this use case, they present three options:

• create a new SSH key

  • not a fan of this because getting the key from the server to my local machine is more involved than the other way around

• move the root SSH key to the non-root user

  • this immediately jumped out as the approach I wanted to use

• startup scripts

  • seems too involved

  • docs say it's best suited for deploying many instances

So I proceed with option #2: move the root SSH key to the non-root user

Check existing keys

The Vultr instructions say to

• make a ~/.ssh folder for the non-root user

• move the authorized_keys file from /root/.ssh/ to /home/<nonRootUser>/.ssh/

• make the non-root user the owner of the /home/<nonRootUser/.ssh directory

...but I think I may have already done some of these steps.

So I log into the Vultr instance as root, and start poking around:

# root@vultr

ls -la ~/.ssh # press enter

# output
drwx------  2 root  wheel  512 May 14 16:15 .
drwx------  3 root  wheel  512 May 14 15:54 ..
-rw-------  1 root  wheel  185 May 14 19:34 authorized_keys

I can see that /root/authorized_keys exists, so I see what's in it:

# root@vultr

cat /root/.ssh/authorized_keys # press enter

# output
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEbll4kwbvR/um3/zzMxEZs7+Jmj8NpAPkqQj8SC6cia idev@MSI
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICaQ9obHa6yYVZPqjQSaeG5OadwxxYguPJVNA7zJpuEm jester@devbox

I see that there are two public keys, one from each of my laptops. I believe these are the keys that I generated and copied over in the last blog post.

To confirm these are the correct keys, I go to each client machine:

# idev@MSI

ls -la ~/.ssh # press enter

# output
drwx------  2 idev idev 4096 May 14 12:15 .
drwxr-x--- 34 idev idev 4096 May 15 18:18 ..
-rw-------  1 idev idev 1342 May 14 11:30 known_hosts
-rw-------  1 idev idev 1120 May 14 11:30 known_hosts.old
-rw-------  1 idev idev  399 May 14 12:11 vultr_ed25519
-rw-r--r--  1 idev idev   90 May 14 12:11 vultr_ed25519.pub

I see what looks like a Vultr public key, so I check its contents:

# idev@MSI

cat ~/.ssh/vultr_ed25519.pub # press enter

# output
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEbll4kwbvR/um3/zzMxEZs7+Jmj8NpAPkqQj8SC6cia idev@MSI

I can see that the public key on the client machine matches what is in the /root/authorized_keys file on the Vultr instance. So far so good.

Now I do the same thing on my other laptop:

# jester@devbox

ls -la ~/.ssh # press enter

# output
drwx------  2 jester jester 4096 May 14 19:37 .
drwxr-x--- 21 jester jester 4096 May 15 18:26 ..
-rw-------  1 jester jester  201 May 14 19:37 config
-rw-------  1 jester jester 1484 May  2 17:22 known_hosts
-rw-------  1 jester jester  506 Apr 29 12:41 known_hosts.old
-rw-------  1 jester jester  399 May 14 19:29 rpi_ed25519
-rw-r--r--  1 jester jester   95 May 14 19:29 rpi_ed25519.pub
-rw-------  1 jester jester  399 May 14 15:33 vultr_ed25519
-rw-r--r--  1 jester jester   95 May 14 15:33 vultr_ed25519.pub

I see an additional key pair and a config file, but right now I'm just concerned with the Vultr key, so I view the contents of the file to see if it matches what's on the server:

# jester@devbox

cat ~/.ssh/vultr_ed25519.pub # press enter

# output
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICaQ9obHa6yYVZPqjQSaeG5OadwxxYguPJVNA7zJpuEm jester@devbox

It does indeed match, so I'm ready to move on to the next step.

Check the non-root user

Recall that the instructions said to make a .ssh folder for the non-root user, but I think I already created it, so I go ahead and check:

# root@vultr

ls -la /home/sharif/.ssh # press enter

# output
drwx------  2 sharif  sharif  512 May 14 15:01 .
drwxr-xr-x  4 sharif  sharif  512 May 14 14:55 ..
-rw-------  1 sharif  sharif  747 May 14 15:04 authorized_keys

Yep, there it is. I suspect the authorized_keys file is not up-to-date because of the configuration changes I made during my last blog post, so I view the contents:

# root@vultr

cat /home/sharif/.ssh/authorized_keys

Sure enough, I've got some old keys that don't match what's on the client machines, so I remove the file:

# root@vultr

rm /home/sharif/.ssh/authorized_keys

Now I can follow Vultr's instructions and move the root user's authorized_keys file over to my non-root user:

# root@vultr

mv /root/.ssh/authorized_keys /home/sharif/.ssh/ # press enter

# make sure it worked

ls -la /home/sharif/.ssh/ # press enter

# output 
drwx------  2 sharif  sharif  512 May 15 22:43 .
drwxr-xr-x  4 sharif  sharif  512 May 14 14:55 ..
-rw-------  1 root    wheel   185 May 14 19:34 authorized_keys
# end output

cat /home/sharif/.ssh/authorized_keys # press enter

# output -- correct keys confirmed
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEbll4kwbvR/um3/zzMxEZs7+Jmj8NpAPkqQj8SC6cia idev@MSI
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICaQ9obHa6yYVZPqjQSaeG5OadwxxYguPJVNA7zJpuEm jester@devbox

Now to adjust the permissions so that the non-root user can properly access the authorized_keys file:

# root@vultr

chown -R sharif:sharif /home/sharif/.ssh # press enter

# make sure it worked
ls -la /home/sharif/.ssh # press enter

# output -- non-root ownership confirmed
drwx------  2 sharif  sharif  512 May 15 22:43 .
drwxr-xr-x  4 sharif  sharif  512 May 14 14:55 ..
-rw-------  1 sharif  sharif  185 May 14 19:34 authorized_keys

The non-root user's configuration looks good; now to test it from both clients.

Test SSH as the non-root user

This part should (hopefully) be pretty painless since I set up SSH configuration files in the last blog post.

Theoretically, I should just be able to change the user from root to sharif in the config, use that config for both laptops, and all should be well.

Let's try it:

# idev@MSI

# /home/idev/.ssh/config
Host myvultrserver.com
    HostName myvultrserver.com
    User sharif
    IdentityFile ~/.ssh/vultr_ed25519

Moment of truth:

# idev@MSI
ssh myvultrserver.com

Success!

The process should be the same on my other laptop:

# jester@devbox

# /home/jester/.ssh/config
Host myvultrserver.com
    HostName myvultrserver.com
    User sharif
    IdentityFile ~/.ssh/vultr_ed25519

# save the config file

# test ssh
ssh myvultrserver.com

Now that I've confirmed I can SSH into the server with both my client machines (laptops), I can finally move on to removing the ability to:

• log in as root

• log in with a password

Disable root and password login

I'll need to log in to my Vultr as root one last time:

ssh root@myvultrserver.com

Now I need to edit the /etc/ssh/sshd_config file to disable root and password login:

# root@vultr

# /etc/ssh/sshd_config

# change 'PermitRootLogin' from 'yes' to 'no'
PermitRootLogin no
# add 'PasswordAuthentication no' to bottom of file
PasswordAuthentication no

Once the file is saved, I need to restart the sshd service on the server:

# root@vultr

rcctl restart sshd # press enter

# output
sshd(ok)
sshd(ok)

Now I can log out of the instance and see if the changes took effect:

# root@vultr
exit # press enter

# idev@MSI
ssh root@myvultrserver.com # press enter

# output
root@sharif.gg: Permission denied (publickey,keyboard-interactive).

This is what I wanted to see -- an error message when trying to log in as root via ssh.

As a final test, I want to see if I can run an elevated command as my non-root user:

# idev@MSI

ssh myvultrserver.com # press enter

# login successful -- sharif@vultr

# attempt to edit a readonly file with sudo
sudo vim /etc/ssh/sshd_config # press enter

# prompted for password
Password: # enter password

sharif is not in the sudoers file. # error
This incident has been reported to the administrator.

I'm feeling pretty good about this, but there is a looming question...

What if I need root?

There is a good chance I'll want to change something else on my server that requires root privileges.

But how do I do that if I just disabled root login and password login?

Thankfully, cloud services like Vultr offer a backdoor through their web console, where you can access a special web terminal with root access.

The web console is usually located in the dashboard/account management area for your cloud provider and is only available once you've logged into your account.

Similarly, you might have multiple devices that need to all log into the same server.

You might even have multiple clients that all need to be able to log into multiple servers, all using SSH with public key authentication -- meaning no password.

This post will guide you through all of the above scenarios, and explain the "need to know" bits of ssh, ssh-keygen, and ssh-copy-id .

If you're already familiar with generating and copying keys, skip to the Multiple Keys on the Client section.

Prework

Note: These instructions assume you are on MacOS, Linux, or WSL

Make sure ssh is installed

First, execute the following command on both the client and the server to make sure ssh is installed:

which ssh

The command above should output a file path.

If not, install ssh:

sudo apt install openssh-client

• after installing, you may need to restart your terminal for the system to recognize ssh as a command

authorized_keys file

On the server, we'll want to check for the ~/.ssh/authorized_keys file. If it doesn't exist, don't worry -- it will be automatically created with the ssh-copy-id command in the following steps

The authorized_keys file contains a list of public keys (one per line). The end of each line shows the username and hostname of the machine from which the public key was generated.

If the file doesn't exist, it just means that you have not yet copied any keys to this server with the ssh-copy-id command.

.ssh folder

On the client, this folder lives in the /home/<yourUserName> directory.

If this folder doesn't exist, it will be created when you generate the key pair with ssh-keygen.

Any private/public key pairs you generate will live in this folder.

Private/public key pairs share the same name; the private key has no extension, and the public key has a .pub extension.

Again, if there are no keys here don't worry.

However, if there are keys here, you may wish to back them up, as there is the potential to overwrite them in the following steps if we're not careful.

Backing up existing keys might look something like this:

mkdir backup_ssh_keys # make a backup folder
cp ~/.ssh -r backup_ssh_keys # copy the ~/.ssh directory to the backup folder
ls -la backup_ssh_keys/.ssh # confirm the keys were copied successfully

ssh-keygen

Generating a key pair is done via the ssh-keygen command.

If you have ssh installed, then you have access to ssh-keygen.

The public/private key pairs are generated using an encryption algorithm. The specifics of the algorithm aren't pertinent to the scope of this blog post. However, the algorithm you supply to the ssh-keygen command could come in handy, depending on your situation.

(More on that later).

The important thing to understand is the relationship between the public and private keys:

I like to think of the public key as a lock on a door, and the private key is what unlocks it.

Check the ~/.ssh folder

ls -la ~/.ssh

If there are no keys in this folder, you can move to the next section.

If there are keys here, take note of the names of the keys. You'll need to reference them in a moment.

Generate the key

If the ssh-keygen command is run without any parameters, it will use the rsa algorithm by default, and generate the keys id_rsa and id_rsa.pub in your ~/.ssh folder.

The algorithm that is used to generate the key pair is also the default name of the key pair.

Using the default key name and location makes it easier to ssh into the server without any additional configuration.

This is why we took down the names of any keys that already existed in our ~/.ssh folder -- if there is already a key named id_rsa in that folder, then we may want to consider using a different algorithm so that we can keep the default name for an easier configuration.

If you're set on using the same algorithm, don't worry -- we'll cover that too.

Generate the key using either ssh-keygen on its own for the defaults, or use ssh-keygen -t <algorithm> to use a specific algorithm (man ssh-keygen to see all algorithm options).

You are then prompted to either accept the default name and path for the file or supply a new one:

Generating public/private rsa key pair.
Enter file in which to save the key (/home/<yourUsername>/.ssh/id_rsa):

If you already have a key named id_rsa in this folder, you'll want to provide a new name for the key here. Otherwise, the old key will be overwritten, and you'll no longer be able to access that server resource until you restore from the backup we created or a new key pair is generated.

If there are no conflict concerns, go ahead and accept the default.

ssh-copy-id

This command is used from the client side to copy the public key over to the authorized_keys file on the server. Just like ssh-keygen, you have access to this command as long as openssh-client is installed.

If you've only got one key pair in your .ssh folder, it's pretty easy.

If you've got more than one key pair, it's still pretty easy (ssh-copy-id uses the most recent key pair in your .ssh folder to determine which key to copy over)

• if it's the most recently-generated key you're wanting to copy over (and you used the default key name and location), you can leave the command as-is

• if not, you can specify the path to the file using the -i flag

Run the command with this syntax:

ssh-copy-id username@server

Again, if you're not wanting to use the most-recently-generated key pair (or need to specify the non-default key name you set), use the -i flag, specifying the file path of the public key like this:

ssh-copy-id -i /home/<yourUsername>/.ssh/<nameOfKey>.pub

If everything worked as expected, you will be prompted for the password for your user on the server, and get a message like this:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'user@server'"
and check to make sure that only the key(s) you wanted were added.

ssh

To doubly confirm that everything worked correctly, let's log out of the server and ssh back into it to make sure we aren't prompted for a password.

exit # log out of the server

# now back on client machine
ssh user@server

You should be logged right in, without being prompted for a password.

If you were prompted, log out of the server again and run the following to see what the issue is:

ssh user@server -v #verbose output for debugging

The most likely cause for this issue is that you need to specify the path to the key file because you supplied a custom filename to ssh-keygen instead of using the default. If that is the case, try running the command with the -i flag, and supply the path to the file:

ssh -i /home/.ssh/<yourCustomKey> user@server

If this works, but you don't want to have to type out the file path each time you access your server, we cover a few options in the next section.

Multiple Keys on the Client

You might need to access multiple servers from the same client machine. That means you need multiple keypairs on the client.

I prefer to be able to log into a server without having to remember the specific key (or path to the key) that I need to use. I want to be able to run ssh user@server every time, and have it work consistently.

For this to work, we have some options:

• if the algorithm used for the key isn't important, simply generate keys for other servers with different algorithms

  • this works if you only need to log into a small number of servers, as there are a limited number of algorithms

• another option is to use a configuration file to specify which key to use for which server

  • more flexible, but requires knowing how the configuration file works

Example with Different Algorithms

From my laptop, I need to access two servers:

• a remote server hosted on vultr

• a Raspberry Pi on my local network

NOTE: for the vultr instance we are using root to keep the example simple. Follow instructions for your cloud provider on setting up SSH keys for a non-root user and then disable root login and password login.

Using the ed25519 algorithm for vultr

First I would generate the key pair for the vultr server using the ed25519 algorithm:

ssh-keygen -t ed25519

I press Enter through all the prompts to accept the default file name and location, choosing not to set a passphrase.

Now I copy the key over to my vultr server with ssh-copy-id

ssh-copy-id root@myvultrserver.com

I get the following output, and am then prompted for the root user's password:

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/idev/.ssh/id_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@myvultrserver.com's password:

I enter my password for the server and get the following output:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@myvultrserver.com'"
and check to make sure that only the key(s) you wanted were added.

Because the default file path was used when generating the key, no file path needs to be specified with the ssh command and I can execute it just like it instructs me to in the output above:

ssh root@myvultrserver.com

I am successfully logged in using public key authentication since I was not prompted for a password.

Using the default rsa algorithm for the Raspberry Pi

Now that I've squared away my ssh configuration to my vultr instance, I generate the key pair for the Raspberry Pi, using the default rsa algorithm

ssh-keygen

Again I accept all the defaults at the prompts.

Since ssh-copy-id uses the most recently generated default identity file, we can use it just like we did for vultr, just replacing the server with the raspberry pi instead:

ssh-copy-id sharif@raspberrypi.local

We get similar output as before; the only difference is the username and host is for the raspberry pi instead of the vultr instance:

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/idev/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
sharif@raspberrypi.local's password:

I enter the password and get the message that the key was added, and am instructed to attempt the public key authentication:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'sharif@raspberrypi.local'"
and check to make sure that only the key(s) you wanted were added.

I type the command as instructed in the output

ssh sharif@raspberrypi.local

I am not prompted for a password once again, meaning the public key authentication was successful.

Example with Configuration File

Let's pretend that any keys generated must use the ed25519 algorithm.

For this example, we'll use the same two servers, but clear out the configuration so we can start fresh:

# as root on vultr server
rm ~/.ssh/authorized_keys

# as sharif on raspberry pi
rm ~/.ssh/authorized_keys

# as idev on client machine
rm ~/.ssh/rd_rsa
rm ~/.ssh/rd_rsa.pub
rm ~/.ssh/id_ed25519
rm ~/.ssh/id_ed25519.pub

Now we can begin generating the keys on the client machine.

Since we know ahead of time that

• we want to use the ed25519 algorithm to generate both key pairs

• the algorithm used to generate the key pair is the default name of the key pair

... we decide that we should not accept the default key pair name when prompted (otherwise the second key would overwrite the first), and give each key pair a more distinguishable name.

Configuration for vultr server

# on client machine
ssh-keygen -t ed25519

This time when we are prompted, instead of accepting the default, we provide a custom file name, but in the default folder:

Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/idev/.ssh/id_ed25519): /home/idev/.ssh/vultr_ed25519

Now we try to copy the key over to the vultr instance, but we get an error

ssh-copy-id root@myvultrserver.com
/usr/bin/ssh-copy-id: ERROR: No identities found #error

This is because we supplied a custom file name, and by default, ssh-copy-id only knows to look for id_rsa, id_ed25519, etc.

To fix this, we use -i and supply our custom key file. Since this is a one-time command, using the -i flag is fine.

ssh-copy-id -i /home/idev/.ssh/vultr_ed25519 root@myvultrserver.com

Now we see our familiar message, and are asked for root@myvultrserver.com's password, and are then instructed to attempt logging in via ssh

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@myvultrserver.com'"
and check to make sure that only the key(s) you wanted were added.

So we try to execute that command:

ssh root@myvultrserver.com

But we're prompted for a password. This is because we need to either:

• supply the identity file with the -i flag again (remember that by default it only knows to look for the keys with the name of the algorithm)

• or

• use a configuration file

Since we are striving for consistency and ease of use, we opt for the configuration file (so we don't have to supply the path of the key want to use every time we ssh into the server).

First we create the configuration file and secure the permissions:

touch ~/.ssh/config && chmod 600 ~/.ssh/config

Now we can open ~/.ssh/config in a text editor and write our first configuration:

Host myvultrserver.com
    HostName myvultrserver.com
    User root
    IdentityFile ~/.ssh/vultr_ed25519

Once we save the file, we can ssh into our server like this:

ssh myvultrserver.com

Since everything is in the configuration file, we just set the user and the file path once, and now we have a convenient way to access the server.

Now we can do the same thing for our Raspberry Pi connection.

Configuration for Raspberry Pi

We follow the same process as we did for vultr, except we name the identity file something else this time.

# on client machine
ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/idev/.ssh/id_ed25519): /home/idev/.ssh/rpi_ed25519

Now we copy the key over to the Raspberry Pi.

ssh-copy-id -i /home/idev/.ssh/rpi_ed25519 sharif@raspberrypi.local

We get the expected output, along with one last password prompt

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/idev/.ssh/rpi_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
sharif@raspberrypi.local's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'sharif@raspberrypi.local'"
and check to make sure that only the key(s) you wanted were added.

Now we can add to the configuration file:

# ~/.ssh/config

Host myvultrserver.com
    HostName myvultrserver.com
    User root
    IdentityFile ~/.ssh/vultr_ed25519

Host raspberrypi.local
    HostName raspberrypi.local
    User sharif
    IdentityFile ~/.ssh/rpi_ed25519

Finally, we access the Raspberry Pi with

ssh raspberrypi.local

Accessing one server from multiple clients

We've already done most of the heavy lifting, and at this point, we just need to connect the dots.

Connecting to multiple servers from one client (which we did in the last section) is more involved than accessing one server from multiple clients since we pretty much just do the same thing on each client.

In fact, if we use the same names for the keys when they are generated on the new client, we can just copy the existing configuration file on each new client.

Let's go through an example.

Example

I just got a new laptop, and I want to be able to access my vultr instance from it.

In the previous examples, the file paths looked like /home/idev/.ssh/...since idev was the name of the user on that computer.

Just for funsies, we'll use ihop as the user on this laptop.

Once I have openssh-client installed on the new laptop, I generate a key, and name it the same thing as I did for the other laptop

# on new laptop
ssh-keygen -t ed25519

# output
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/ihop/.ssh/id_ed25519): /home/ihop/.ssh/vultr_ed25519 # same key name, but on a different machine

Now our normal song and dance of copying the key over

ssh-copy-id -i /home/ihop/.ssh/vultr_ed25519 root@myvultrserver.com

Then we make our new config file and adjust the permissions

touch ~/.ssh/config && chmod 600 ~/.ssh/config

And now we can simply copy the same configuration from our other laptop, since the configuration is using relative paths, and does not specify a username

# ~/.ssh/config

Host myvultrserver.com
    HostName myvultrserver.com
    User root
    IdentityFile ~/.ssh/vultr_ed25519

Now we can use the same command (ssh myvultrserver.com) on both laptops and it works the same way.

This pattern can be followed for each new client machine that needs to access the server.

Access multiple servers from multiple clients

Similar to the last section, this sounds more complicated than it is. You may have already figured it out if you've read this far.

In this last section, we accessed the same server from a new client machine by generating a new key with the same name, then copying the existing configuration to the new client machine.

As it stands, we have two clients accessing the vultr server, but what if we also wanted the new laptop to have access to the Raspberry Pi?

Example

We follow the same steps as the last section, except we use the name of the first Raspberry Pi key we created instead of the vultr key:

# on new laptop
ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/ihop/.ssh/id_ed25519): /home/ihop/.ssh/rpi_ed25519

And copy the key

ssh-copy-id -i /home/ihop/.ssh/rpi_ed25519 sharif@raspberrypi.local

And update our configuration file (it's now identical to the one on the idev laptop):

# ~/.ssh/config

Host myvultrserver.com
    HostName myvultrserver.com
    User root
    IdentityFile ~/.ssh/vultr_ed25519

Host raspberrypi.local
    HostName raspberrypi.local
    User sharif
    IdentityFile ~/.ssh/rpi_ed25519

Apply this same pattern for as many clients and servers as you need.

I still barely know what I'm doing, but I know enough now to fix the initial issues I was having, so now I can move on to doing cool shit with it.

We're going to go over each of the issues I had, how I solved them, and finally -- make an LED blink using JavaScript.

Issues Out of the Box

My unit wasn't defective or anything. These were software issues. Still, I wasn't sure how to solve them; my normal approaches weren't working.

The first issue was that Bluetooth wouldn't work. Not that big of a deal, since I had a wired keyboard and mouse laying around. Still kind of annoying, but I was willing to make peace with it since it's a barebones computer.

The next issue was a bit more problematic, though. I was getting errors any time I tried to use apt or apt-get. Everything I came across online suggested I update the distro mirror URL in the /etc/apt/sources.list file.

I tried for a while to make this work, but it just wasn't happening. After I changed up my Google search a bit, I discovered that I was running an older distro, Stretch. All of the forums were referencing newer distros -- Buster and Bullseye.

This gave me a bit more hope. Upgrading the distro would surely fix my issues.

Distro Updates

From a forum answer, I gleaned that Buster came after Stretch, and Bullseye came after Buster. It was recommended to first upgrade to Buster, then to Bullseye, rather than upgrading from Stretch straight to Bullseye.

Again, the forums suggested editing the /etc/apt/sources.list file, but this time instead of adding a new mirror URL, the suggestion was to change any instances of the word 'stretch' to 'buster', then run a sudo apt-get update.

This worked, but the next time I booted up the pi, I got an error saying that the root account was locked. I couldn't even access the desktop.

Googling some more, I discovered the Raspberry Pi Imager tool, which is what I should have been using from the start. At this point, I was glad I had the Cana Kit, because it came with a microSD caddy. Now all that was left to do to do was:

• put the microSD card from the pi into the caddy

• plug the caddy into my laptop

• open the imager tool

• select the Bullseye distro

• select the USB caddy

From here, the imager tool took care of the rest. I popped the microSD back into the pi, and Bullseye loaded right up.

Once Bullseye was installed, I was able to connect to Bluetooth, and all of my shell commands worked the way I expected them to. With that, I proceeded to install zsh, oh-my-zsh, and neovim.

Hardware Time

Now that my software issues were out of the way, it was finally time to work on the hardware. Well, almost.

The literature that came with my CanaKit had examples of setting up a circuit and then running it on the Pi using Python. I've used Python before and was confident I could get it to work in Python, but I am a big fan of sticking to one programming language for as many things as possible.

Seeing that JavaScript is ubiquitous these days, I figured there had to be some resources out there to guide me in that direction.

Sure enough, I was in luck. The first steps were to install Node.js and npm. That was simple enough, but I wasn't sure how Node could let me control the hardware. Then I realized, "Well shit, I don't actually know much about this hardware at all".

That was when I found this blog post, which gave me a primer on how the pins on the pi work, the reasoning behind using resistors, and a few other electrical engineering tidbits.

The blog post also filled in my knowledge gap around how Node can interface with the Raspberry Pi -- through the onoff package. onoff implements the GPIO logic of the pins in Node.js, so that people like me who find that sort of thing intimidating can work with a simple/intuitive API and have fun with their pi (hey, that rhymes!)

Although the blog post had an example with an LED using pin 4, I didn't want to copy it directly. I always like to change things up a bit to make sure I understand the concepts, instead of just following someone else's work to the tee.

So what I decided to do was use the Python example from the CanaKit (which used pin 18), and convert the Python code they provided into Node.js code. Once the circuit was set up and the code was written, running it was as easy as node led.js.

I was pleasantly surprised at how straightforward the coding part of this was. I had expected to install a bunch of things to get it working.

Extras

At this point, I was feeling pretty good. I felt like I finally had a basic understanding of everything that came in my CanaKit, except for one box that said "breakout board" on it.

Off to the Google Machine again. Turns out, a breakout board is just a labeled (and therefore more convenient) set of pins for the Raspberry Pi.

When I was going through the LED example in the previous section, I had to keep referencing a GPIO chart that tells you which pin is which. The breakout board has labels next to each of the 40 pins so that you don't need to reference a chart.

The tradeoff here is the form factor. The original 40 pins fit right in the tiny Raspberry Pi case. The breakout board attaches to the original 40 pins and sits outside the case. If the pins inside the Raspberry Pi were labeled, then the device would have to be made bigger to account for that.

Next Steps

This has got me excited about the possibilities of tinkering with hardware. I want to see how far I can take it, both on the hardware and the software side. How complex of a circuit can I make? What sort of useful hardware can I wire this up to? Can I have that hardware signal some software I write, based on the conditions of the circuit?

I hope you'll stick around to find out!

Access tokens are part of the authentication flow for accessing a resource that lives within Salesforce. Think of it as your way of proving that you are allowed to have access to that resource.

For example, let's say we have a @RestResource REST API endpoint that we have created using Apex. We want to be able to consume the data from that REST API from an external web application.

The web application has to prove that it has access to the Salesforce environment to access the data behind the REST API. This is where access tokens come in.

Many different authorization flows in Salesforce use access tokens. This walkthrough will focus on the username/password auth flow. This is not a viable workflow for a production environment. However, it is a good workflow for testing out HTTP requests quickly.

Postman

Postman is a piece of software that provides a user interface for retrieving access tokens and making HTTP requests.

In this walkthrough, we make use of the Salesforce Platform APIs Collection within Postman, which makes getting an access token relatively straightforward.

We can think of a Postman Collection as a group of pre-configured HTTP requests (GET, POST, etc.)

As such, the Salesforce Platform APIs Collection has many pre-configured requests. However, we are mainly concerned with the access token request.

The steps below will illustrate:

• Adding the Salesforce Platform APIs Collection to Postman

• Making the access token request

• Using the access token

• Troubleshooting for several scenarios

1. Postman Log in / Sign Up

• Head over to postman.com (or use the desktop app)

• Log in to your Postman account (or create a new one)

2. Add and Fork the Collection

• At the top of the page, click the search box

• Search Salesforce Platform APIs

• Click on the collection

• Press the 'Fork' button to fork the collection

• Give the fork a name

• Select a Workspace for the fork

• Press the 'Fork Collection' button

The collection should now be open in your workspace.

3. Request Access Token

Your forked collection should have opened to the 'Authorization' tab.

• Scroll down to the bottom of the Authorization tab

• Leave all default values

• Press the 'Get New Access Token' button

  

• The Salesforce login page will appear in a new window

  • If you are not presented with a standard username/password prompt, see the Troubleshooting section below

• Enter your Salesforce username and password

• Log in

• You should receive the following prompt

• Click 'Allow'

• You will be redirected back to Postman, and should see the following screen

  

• Copy the access token to your clipboard so that we can use it in the next step

• Click the 'Use Token' button

4. Use the Access Token

Now that we have the token, we can use it in a request.

Construct request URL

1. Get your base URL for the request

   

2. Add https:// to the beginning of the base URL

3. Add /<your_api_endpoint_route> to the end of the base URL

For our example, we are using a @RestResource that lives at /services/apexrest/Contacts

Therefore, the full request URL we end up with is:

https://saleshorse-dev-ed.develop.my.salesforce.com/services/apexrest/Contacts

Create a Postman Collection

Chances are, you're going to want to make this same request (or a very similar one) at a later date. And since this request is separate from the access token request, we can store it in a separate Collection.

1. Click the 'Collections' tab in Postman, then click the '+' button

   

2. Click the pencil icon to rename the Collection

   

3. Select 'OAuth 2.0' under the 'Type' dropdown

   

4. Scroll down and paste the access token into the 'Token' field, being sure there is no whitespace

   

5. Add a request to the collection by expanding the collection in the sidebar and clicking the 'Add a request' link (or by pressing the three dots (...) and selecting 'Add request')

   

6. Rename the request from the default 'New Request'

   

7. Select the 'Authorization' tab, and ensure that 'Inherit auth from parent' is selected as the 'Type'. (Because we entered the token for the entire collection, we don't have to enter it again for each request within that collection)

   

8. Paste in the request URL and press 'Send'

   

9. The request result appears at the bottom of the window

   

10. Save the Collection

    

Troubleshooting

Trouble getting access token

If a list of saved usernames appears, click the 'Log in with a Different Username' link.

• The Salesforce login page will appear in a new window

  • If the URL in the new window starts with login.salesforce.com

    • Click the 'Use Custom Domain' link

      

  • If the URL in the new window is the URL of your target Salesforce org, skip this step and enter your username and password (the 'Use Custom Domain' link will not be present)

Find your domain URL

• Log into the Salesforce org that you are requesting an access token for

• Click on your avatar in the top-right

• Copy the URL up until the .my.salesforce.com

  

• Paste it into the 'Custom Domain' field (Salesforce automatically adds the .my.salesforce.com

  

• Press 'Continue'

• Log in with your username and password

Trouble using access token

If you receive the following error when using the access token in a request, follow the steps below to resolve the issue:

{
    "message": "Session expired or invalid",
    "errorCode": "INVALID_SESSION_ID"
  }

Request another token

Access tokens are only valid for a limited time, so you may just be seeing this message because you need to request a fresh token to use in your request.

Check that the token was copied and pasted correctly

You may have missed a character when highlighting the access token. Request a new token, and make sure that is not the issue.

If you are sure you have copied the token correctly, check that there are no line breaks or leading/trailing spaces in the access token.

In the screenshot above, Postman is trying to hint to us that we have a line break at the end of our token, by displaying a carriage return symbol (highlighted in red).

To fix this, place your cursor on the empty line below the token, then press the backspace key. You should see the token input shrink to fit the length of your token, and the carriage return symbol should go away:

Make sure the token was requested for the correct Salesforce org

TL;DR -- log out of any logged-in Salesforce sessions, and request the token again

If you frequently log into more than one Salesforce org, you may run into this issue.

When you log into a Salesforce org, it creates a session within your browser. This is why you can click a link for a record in that Salesforce org and be taken straight to the record, without having to log in.

If you are using the Postman web application in the same browser session as the Salesforce org you are logged into, Postman is going to request the access token for the org you're already logged into.

If the org you're already logged into isn't the org you're requesting the access token for, this isn't the behavior you want.

If you're still scratching your head, let's work through an example:

Say your company uses Salesforce internally for things like showing company news and events. You logged into the internal Salesforce org when you started your day, looked over the announcements, then started working on integrating a REST API endpoint for a client, which lives in the client's Salesforce org.

When you request the access token with Postman through the Postman web app in the browser, the steps go something like this:

1. Postman requests an access token from the Salesforce access token endpoint. (Note that this endpoint is not specific to any one org)

2. Salesforce says "I might have a token for you, but first I need you to tell me who you are", and attempts to relay this message back to Postman through the browser

3. But before that "who are you?" message can even get back to Postman your browser session butts in and says, "No need! Check it out, they're already logged in."

4. Salesforce says "Whatever. Here's your token."

5. You are then issued a token for the incorrect org, because you never got the chance to specify which org you wanted the token for

To get around this issue, you can either:

1. Log out of the Salesforce org in your current browser session

2. Go to Postman in a private/incognito browser window to force a new session

After picking one of the two options, request a new token via Postman, and enter the credentials for the desired org when prompted.