[GUIDE] How to build lbrynet on FreeBSD [Step by step]

rapydi · August 7, 2024, 2:11pm

I can’t edit posts on this forum, so you might find later updates of this guide here. If you have any issues following the instructions, feel free to comment below.

This has been tested on FreeBSD 14.1-RELEASE amd64.

WARNING: At the time of writing, the latest lbry-sdk release is v0.113.0, which needs Python 3.7 due to a bug. Python 3.7 is EOL. lbry-sdk requires older dependencies & they might have vulnerabilities. Please use security measures to protect your system if that’s important to you.

I figured it would be a good idea to keep lbry-sdk or lbrynet dependencies independent of the system /usr/local. This way we don’t have to mess with anything already installed and removal would be easier (if you desire to do so later). So after building from source, installations will go to /usr/local/lbdeps and they’ll have -Wl,R or -Wl,rpath linking option added to them so that setting LD_LIBRARY_PATH environment variable is not necessary every time using them. So we need to build from sources. Don’t worry. These instructions are written so that almost anyone can follow.

I tried to make this easy. The other alternative would be to manually go through revisions in FreeBSD Ports source code for multiple packages, which might be even harder to follow. But let me know if this can be made easier.

If this sounds a bit hard to follow, you can download a compiled copy of lbrynet for FreeBSD.

For laptops: If your machine is getting very hot during builds, try TuningPowerConsumption FreeBSD wiki page. The page is old, but most of the things work. In my experience, it makes the machine efficient and causes less heat. Make sure to keep backups before editing any files so that if things go wrong you can recover them from single user mode. If it fails, try cleaning the insides (if you are allowed/able to) or place a table fan towards the machine’s hottest place. Modern machines usually take care of heat management by themselves (ACPI), but I like to be safe since fan configuration might not be perfect.

Step 0: Basic prep

Here are some basic packages needed:

$ doas pkg install git-lite python3 gnupg gcc gmake libtool pkgconf autoconf automake

I’m using doas, but sudo might also work. No matter which one you use, be sure to configure it for your nonroot user first. Also, my shell is bash.

Prepare dir for sources:

$ cd ~
$ mkdir lbrepos

Step 1: Building OpenSSL 1.x from source

lbrynet 0.113.0 requires Python cryptography==2.5, for which, we need OpenSSL 1.x. 1.1.1w is the latest 1.x version as of writing this.

$ cd ~/lbrepos
#~ OpenSSL 1.1.1w is latest atm. Later versions might also work, but not tested.
$ git clone --depth 1 -b OpenSSL_1_1_1w https://github.com/openssl/openssl
$ cd openssl
$ ./config -fPIC -shared --prefix=/usr/local/lbdeps
$ gmake  # GNU Make is gmake on FreeBSD
$ doas gmake all install_sw

Step 2: Install Python 3.7

We’re going to use a separate sqlite install for python to use to get around an error I faced. Relax, we’re going to use amalgamation autoconf source code to make this super easy:

$ cd ~/lbrepos
#~ SQLite 3.46.0 is latest atm. Later versions might also work, but not tested.
$ curl -LO https://sqlite.org/2024/sqlite-autoconf-3460000.tar.gz
$ tar -xvf sqlite-autoconf-3460000.tar.gz
$ cd sqlite-autoconf-3460000
$ ./configure --prefix=/usr/local/lbdeps
$ gmake
$ doas gmake install

To test (optional):

$ echo .version | /usr/local/lbdeps/bin/sqlite3
#~ It should say "SQLite version 3.46.0"
$ echo .dbconfig | /usr/local/lbdeps/bin/sqlite3 | grep load_extension
#~ In the above output "load_extension" should be "on" if you want to use it
#~ with "--enable-loadable-sqlite-extensions" python configuration option.

Let’s install Python 3.7:

$ cd ~/lbrepos
$ PYVER=3.7.17
$ curl -LO https://www.python.org/ftp/python/$PYVER/Python-$PYVER.tar.xz.asc
$ curl -LO https://www.python.org/ftp/python/$PYVER/Python-$PYVER.tar.xz
#~ Got the gpg fingerprint below from https://www.python.org/downloads/ under
#~ "OpenPGP Public Keys" heading, which also links to https://keybase.io/nad/
$ gpg --keyserver hkps://keys.openpgp.org --recv-keys 2D347EA6AA65421D FB9921286F5E1540
$ gpg --verify Python-$PYVER.tar.xz.asc
#~ If it says 'Good signature from "Ned Deily (Python release
#~ signing key) <nad@python.org>"' then it's a good copy.
#~ If it doesn't say above, discard the copy and download again.

$ tar -xvf Python-$PYVER.tar.xz
$ cd Python-$PYVER

#~ For "No module named '_ctypes'" error we need libffi and --with-system-ffi etc.
#~ https://stackoverflow.com/questions/55867213/building-python3-7-3-from-source-missing-ctypes
#~ PKG_CONFIG_PATH might be needed which is also covered later
#~ https://github.com/pyenv/pyenv/issues/1183#issuecomment-421401095
$ doas pkg install libffi

Now to build:

#~ You can add "--enable-optimizations" to the following. It will take some
#~ extra time to build, but resulting python "can significantly enhance the 
#~ execution speed of Python programs" (https://dnmtechs.com/understanding-the-purpose-of-enable-optimizations-flag-in-python-3-compilation/)
$ ./configure --enable-shared --enable-loadable-sqlite-extensions \
--with-system-ffi --with-ensurepip=install --prefix=/usr/local/lbdeps \
--with-openssl=/usr/local/lbdeps \
CFLAGS="-I/usr/local/lbdeps/include -I/usr/local/include/xorg \
-I/usr/local/include" \
LDFLAGS="-L/usr/local/lbdeps/lib -Wl,-R/usr/local/lbdeps/lib \
-L/usr/local/lib/xorg -Wl,-R/usr/local/lib/xorg -L/usr/local/lib \
-Wl,-R/usr/local/lib" \
PKG_CONFIG_PATH="/usr/local/lbdeps/lib/pkgconfig:/usr/local/libdata/pkgconfig"
$ gmake
$ gmake test  # optional
$ doas gmake altinstall

Test:

$ /usr/local/lbdeps/bin/python3.7 --version
Python 3.7.17
$ /usr/local/lbdeps/bin/python3.7 -c \
'import sqlite3; print(sqlite3.sqlite_version)'
3.46.0
$ /usr/local/lbdeps/bin/python3.7 -c 'import ssl; print(ssl.OPENSSL_VERSION)'
OpenSSL 1.1.1w  11 Sep 2023

Step 3: Prepare lbry-sdk venv

$ cd ~/lbrepos
$ git clone --depth 1 -b v0.113.0 https://github.com/lbryio/lbry-sdk.git
$ cd lbry-sdk
$ /usr/local/lbdeps/bin/python3.7 -m venv lbry-venv
$ source lbry-venv/bin/activate

This will add a (lbry-venv) prefix to your shell prompt. It means you are in a venv with python 3.7 being the default python. The bug I mentioned at the beginning is the reason why we need to follow the build process in a python 3.7 venv. Don’t exit this venv unless the guide tells you later.

$ pip install --upgrade pip  # upgrade pip to get rid of messages
$ python --version  # it should say "Python 3.7.17"

Step 4: `libtorrent` Python Bindings

Without quitting venv:

$ cd ~/lbrepos
#~ Used 1.84.0 here. Later versions might also work, but not tested.
$ curl -LO https://github.com/boostorg/boost/releases/download/boost-1.84.0/boost-1.84.0.tar.xz
$ tar -xvf boost-1.84.0.tar.xz
$ cd boost-1.84.0
#~ Very IMPORTANT for building libtorrent
$ export BOOST_ROOT=$PWD  # the directory with directories like libs, more, status
$ export BOOST_BUILD_PATH=$BOOST_ROOT/tools/build

Install b2 from source to be used later on:

$ cd "$BOOST_BUILD_PATH"
$ ./bootstrap.sh
#~ So that we can use b2 binary that we just built without installing on system
$ export PATH="$BOOST_BUILD_PATH:$PATH"
#~ The following output should not show b2 from /usr/local, but from ~/lbrepos
$ which b2
/home/username/lbrepos/boost-1.84.0/tools/build/b2

Now create a ~/user-config.jam and put:

using python : 3.7 : /usr/local/lbdeps/bin/python3.7 : /usr/local/lbdeps/include/python3.7m : /usr/local/lbdeps/lib/python3.7 ;
using gcc : libtorrent : g++ : <cxxflags>-fext-numeric-literals <compileflags>-fPIC <compileflags>-I/usr/local/lbdeps/include <linkflags>-L/usr/local/lbdeps/lib <linkflags>-Wl,-R/usr/local/lbdeps/lib <linkflags>-Wl,-rpath=/usr/local/lib/gcc13 ;

When gcc --version is run on my system it shows gcc (FreeBSD Ports Collection) 13.2.0, so I ran pkg info -D gcc13 and got instructions on adding -Wl,-rpath=/usr/local/lib/gcc13 link option. If you have a different version, change it accordingly in above file.

Now so that running doas b2 finds this same .jam file, we need to have the same file as /root/user-config.jam:

$ doas ln -s /home/$USER/user-config.jam /root/user-config.jam

Let’s build boost libraries needed and install them:

$ cd "$BOOST_ROOT"
$ b2 --with-system --with-python
$ doas b2 --with-system --with-python install --prefix=/usr/local/lbdeps

Now to libtorrent python bindings:

$ cd ~/lbrepos
#~ tar download has all submodules populated - more convenient
#~ libtorrent version 2.0.6 is reuired by lbry-sdk 0.113.0
$ curl -LO https://github.com/arvidn/libtorrent/releases/download/v2.0.6/libtorrent-rasterbar-2.0.6.tar.gz
$ tar -xvf libtorrent-rasterbar-2.0.6.tar.gz
$ cd libtorrent-rasterbar-2.0.6

We need a fix for a rule "version.boost-build" unknown error. Save this in a file called myfix.patch:

diff --git a/Jamfile b/Jamfile
index a8c476b..5039986 100644
--- a/Jamfile
+++ b/Jamfile
@@ -22,6 +22,8 @@ ECHO "CXXFLAGS =" $(CXXFLAGS) ;
 ECHO "LDFLAGS =" $(LDFLAGS) ;
 ECHO "OS =" [ os.name ] ;
 
+jam-version = [ modules.peek : JAM_VERSION ] ;
+
 if $(BOOST_ROOT)
 {
        ECHO "building boost from source directory: " $(BOOST_ROOT) ;
@@ -163,10 +165,11 @@ rule linking ( properties * )
                # which only works on ELF targets with gcc
                result += <linkflags>-Wl,--export-dynamic <linkflags>-rdynamic ;
        }
-       else
+       else if [ version.version-less $(jam-version) : 1990 0 ]
        {
-               # backtraces don't work with visibility=hidden, so we only add that in
-               # the else-block
+               # the visibility feature was introduced in boost-1.69. This was close to
+               # when the verisoning scheme changed from year to (low) version numbers.
+               # in boost-1.70
                result += <visibility>hidden ;
        }
 
@@ -940,8 +943,10 @@ rule install-paths ( properties * )
        # package.paths was introduced in boost-1.70 (2018.02)
        # however, boost build's versioning scheme changed in boost-1.71 to version
        # 4.0
-       local boost-build-version = [ SPLIT_BY_CHARACTERS [ version.boost-build ] : "-" ] ;
-       if [ version.version-less [ SPLIT_BY_CHARACTERS $(boost-build-version[1]) : "." ] : 2018 03 ]
+       # so, if versions are 4.0+ we want to use package.paths, but if it's a year,
+       # say 2018, that means it's old and we use the fallback below. Any version <
+       # 1990 is considered the 4.0 and later numbering scheme.
+       if [ version.version-less 1990 0 : $(jam-version) ]
        {
                import option ;
                import property ;

Now apply the patch:

patch < myfix.patch

Build + install libtorrent and then libtorrent python bindings:

$ b2 variant=release library-path=/usr/local/lbdeps/lib
$ doas b2 variant=release library-path=/usr/local/lbdeps/lib install --prefix=/usr/local/lbdeps
#~ Install Python module
$ python setup.py build_ext --b2-args="toolset=gcc-libtorrent variant=release \
libtorrent-link=shared boost-link=shared library-path=/usr/local/lbdeps/lib" install

Confirm if libraries (.so files) are taken from /usr/local/lbdeps:

$ ldd $VIRTUAL_ENV/lib/python3.7/site-packages/*/libtorrent.so
...
	libboost_python37.so.1.84.0 => /usr/local/lbdeps/lib/libboost_python37.so.1.84.0 (0xfcb1cbb5000)
	libtorrent-rasterbar.so.2.0.6 => /usr/local/lbdeps/lib/libtorrent-rasterbar.so.2.0.6 (0xfcb20800000)
	libboost_system.so.1.84.0 => /usr/local/lbdeps/lib/libboost_system.so.1.84.0 (0xfcb1b6f3000)
	libthr.so.3 => /lib/libthr.so.3 (0xfcb1c578000)
	libutil.so.9 => /lib/libutil.so.9 (0xfcb1d604000)
	libssl.so.1.1 => /usr/local/lbdeps/lib/libssl.so.1.1 (0xfcb1ea81000)
	libcrypto.so.1.1 => /usr/local/lbdeps/lib/libcrypto.so.1.1 (0xfcb1e1c0000)
	libstdc++.so.6 => /usr/local/lib/gcc13/libstdc++.so.6 (0xfcb23e00000)
	libm.so.5 => /lib/libm.so.5 (0xfcb1eb9c000)
	libgcc_s.so.1 => /usr/local/lib/gcc13/libgcc_s.so.1 (0xfcb1f9ff000)
	libc.so.7 => /lib/libc.so.7 (0xfcb1b134000)

libboost_python37.so, libtorrent-rasterbar.so, libboost_system.so, libssl.so, libcrypto.so should be from /usr/local/lbdeps. If the output does not show the paths like above, check the user-config.jam file to see if the paths are correctly set. You might have to repeat some of the commands.

Step 5: Using lbrynet

Before building, let’s mod some files for FreeBSD:

$ cd ~/lbrepos/lbry-sdk
$ sed -i'.bak' -e "s/elif 'linux' in sys/elif 'linux' or 'bsd' in sys/" lbry/conf.py
$ sed -i'.bak' -e \
's/if d\["os_system"\] == "Linux":/if d\["os_system"\] == "Linux" or d\["os_system"\] == "FreeBSD":/' \
lbry/extras/system_info.py

To build lbrynet:

$ gmake install

Now test with:

$ lbrynet --version
lbrynet 0.113.0

The lbrynet that you used above is not a binary. If you run cat $(which lbrynet) you will see that it’s just a python script. So it requires to have the whole lbry-sdk source code to be present. You will notice that it has the lbry-sdk source path in its first line (shebang line). The official lbry-sdk project provides a single binary which is more convenient. There are instructions on that below.

If you don’t want to go the extra mile of creating a single lbrynet binary, then you should keep the ~/lbrepos/lbry-sdk in that place and you can copy and use ~/lbrepos/lbry-sdk/lbry-venv/bin/lbrynet anywhere you want on the system. As long as the the lbry-sdk source is on the directory mentioned in its first line, it should work. For example, you can use the file with FastLBRY GTK by copying it as ...FastLBRY-GTK/flbry/lbrynet.

Before starting lbrynet daemon, you might want to change the default wallet servers because those don’t often work. Working servers might change in future. For now I have it working like this below. Optionally, uncomment and set download_dir to save downloaded content in your preferred place.

You can save something like below as ~/.local/share/lbry/lbrynet/daemon_settings.yml:

blockchain_name: lbrycrd_main
#download_dir: /home/user/Downloads/LBRY_Downloads
lbryum_servers:
- c-hub1.odysee.com:50001
- a-hub1.odysee.com:50001
save_blobs: true
save_files: false
share_usage_data: true
tcp_port: 3333
udp_port: 4444
use_upnp: true

You may find the need to login to lbrynet before using it. This login is different from your browser sessions like in Odysee.com. So even if you are already logged in to odysee.com, you’ll have to login to lbrynet separately, since your wallet, channels etc. will not be available otherwise.

To do that without LBRY Desktop is a bit harder. There is a script (also posted here) for this purpose. Don’t forget to read the comments at the beginning, esp. the warnings. To continue:

Run /path/to/lbrynet start, let it run for some time until it initializes.
Save the file as signin-lbrynet.py.
Then on another terminal run python3 signin-lbrynet.py and follow the on screen instructions. If successful, it will sync and make available the wallets and channels, so I keep it running until “Sync” related messages stop appearing in ~/.local/share/lbry/lbrynet/lbrynet.log.
Then when done, go back to the shell where you started lbrynet and press Ctrl+C to stop it.

Step 6: Creating a single `lbrynet` binary (optional, but convenient)

We’ll have to do this with PyInstaller. PyInstaller dropped support for python 3.7 from 6.x. 5.13.2 is the last 5.x version.

$ pip install pyinstaller==5.13.2
#~ to create a single "lbrynet" file:
$ cd ~/lbrepos/lbry-sdk
$ pyinstaller --onefile --strip --name lbrynet lbry/extras/cli.py

--strip is optional here to keep the file sizes low. It gets rid of information from the binary file that is needed for debugging (important for developers and troubleshooting, not so much for average users). A file lbry-sdk/dist/lbrynet should be created with file size around 15MB (~24MB not stripped).

Now you can run deactivate from the venv to quit out of it.

This is optional, but you can copy lbrynet to system /usr/local/bin so that you can run it from any directory from terminal, like:

$ doas cp dist/lbrynet /usr/local/bin/

You are now ready to keep using it. If everything went as expected, you can now delete (if you wish) /usr/local/lbdeps dir and the single dist/lbrynet file should still work. You can copy the file somewhere else and also delete (if you wish) ~/lbrepos dir.

Removal (optional)

If you don’t like the outcomes of something and decide to remove everything you built, just delete the two output directories ~/lbrepos and /usr/local/lbdeps. Easy. Doing all the hard work to keep lbrynet related installations separate from usual /usr/local dir has this benefit. You might also want to remove ~/user-config.jam and /root/user-config.jam. Also look into the usual files/dirs that lbrynet creates, such as, ~/.local/share/lbry and the path you set on download_dir on daemon_settings.yml.

License: This guide is made available under CC0 1.0 Universal. Feel free to use it, modify it, relicense it, without asking for permission.