Knowledge Center

Popular Search themodesp32sliesp
Categorieën bekijken

nVidia Jetson NANO – Prepare for AI tools and LLM

Table of Contents

Official image with UI #

You can download Jetson IMG files wich you can burn on a SD card here: https://developer.nvidia.com/embedded/downloads/

Search for “Jetson Nano Developer Kit SD Card Image“:


https://developer.nvidia.com/embedded/l4t/r32_release_v7.1/jp_4.6.1_b110_sd_card/jeston_nano/jetson-nano-jp461-sd-card-image.zip

or always the latest version:
https://developer.nvidia.com/jetson-nano-sd-card-image

Prepare software with APT #

Install Python3, build tools, AI LLM libraries and dependancies:

sudo apt-get update && sudo apt-get install -y python3 python3-pip python3-venv python3-dev python3-skimage python3-packaging python3-tqdm python3-filelock python3-pil libopenblas-base libopenblas-dev libopenmpi-dev libomp-dev liblapack-dev libjpeg-dev libffi-dev libncurses5-dev libncursesw5-dev libreadline-dev libsqlite3-dev libgdbm-dev libdb5.3-dev libbz2-dev libexpat1-dev libssl-dev zlib1g-dev pkg-config gfortran nano curl git wget cmake build-essential

Set Python3 as default:

sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 1

Output:

update-alternatives: using /usr/bin/python3 to provide /usr/bin/python (python) in auto mode

Check with:

python --version

Output:

Python 3.6.9

Prepare software with PIP3 #

Install these modules thhrough the PIP system, them need compilation:

sudo -H pip3 install setuptools-rust importlib-metadata sacremoses // huggingface_hub

Update the setuptools/wheel #

Ubuntu 18.04 has some old setuptools/wheel, update these with this command

sudo -H pip3 install --upgrade "setuptools<60" wheel

Output:

Collecting setuptools<60
  Downloading https://files.pythonhosted.org/packages/b0/3a/88b210db68e56854d0bcf4b38e165e03be377e13907746f825790f3df5bf/setuptools-59.6.0-py3-none-any.whl (952kB)
    100% |████████████████████████████████| 962kB 515kB/s
Collecting wheel
  Downloading https://files.pythonhosted.org/packages/27/d6/003e593296a85fd6ed616ed962795b2f87709c3eee2bca4f6d0fe55c6d00/wheel-0.37.1-py2.py3-none-any.whl
Installing collected packages: setuptools, wheel
  Found existing installation: setuptools 39.0.1
    Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
  Found existing installation: wheel 0.30.0
    Not uninstalling wheel at /usr/lib/python3/dist-packages, outside environment /usr
Successfully installed setuptools-59.6.0 wheel-0.37.1

Rust programming language/compiler #

Rust is blazingly fast and memory-efficient: with no runtime or garbage collector, it can power performance-critical services, run on embedded devices, and easily integrate with other languages.

Some AI LLM libraries like safetensors and tokenizer need RUST to compile.

Install rust with this command:

curl https://sh.rustup.rs -sSf | sh

Choose 1

   default host triple: aarch64-unknown-linux-gnu
     default toolchain: stable (default)
               profile: default
  modify PATH variable: yes

1) Proceed with standard installation (default - just press enter)
2) Customize installation
3) Cancel installation
>1

Output:

info: profile set to 'default'
info: default host triple is aarch64-unknown-linux-gnu
info: syncing channel updates for 'stable-aarch64-unknown-linux-gnu'
info: latest update on 2025-10-30, rust version 1.91.0 (f8297e351 2025-10-28)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
 20.5 MiB /  20.5 MiB (100 %)  10.2 MiB/s in  2s
info: downloading component 'rust-std'
 27.0 MiB /  27.0 MiB (100 %)  10.1 MiB/s in  3s
info: downloading component 'rustc'
 58.3 MiB /  58.3 MiB (100 %)  10.1 MiB/s in  7s
info: downloading component 'rustfmt'
info: installing component 'cargo'
  9.7 MiB /   9.7 MiB (100 %)   6.8 MiB/s in  1s
info: installing component 'clippy'
info: installing component 'rust-docs'
 20.5 MiB /  20.5 MiB (100 %)   2.4 MiB/s in 12s
info: installing component 'rust-std'
 27.0 MiB /  27.0 MiB (100 %)   6.0 MiB/s in  8s
info: installing component 'rustc'
 58.3 MiB /  58.3 MiB (100 %)   5.5 MiB/s in 12s
info: installing component 'rustfmt'
info: default toolchain set to 'stable-aarch64-unknown-linux-gnu'

  stable-aarch64-unknown-linux-gnu installed - rustc 1.91.0 (f8297e351 2025-10-28)


Rust is installed now. Great!

To get started you may need to restart your current shell.
This would reload your PATH environment variable to include
Cargo's bin directory ($HOME/.cargo/bin).

To configure your current shell, you need to source
the corresponding env file under $HOME/.cargo.

This is usually done by running one of the following (note the leading DOT):
. "$HOME/.cargo/env"            # For sh/bash/zsh/ash/dash/pdksh
source "$HOME/.cargo/env.fish"  # For fish
source $"($nu.home-path)/.cargo/env.nu"  # For nushell

For compiling stuff later-on we need a specific version of rust (not to new) to compile succesfully, revert the version to 1.72.0

source $HOME/.cargo/env
rustup install 1.72.0

Set as default:

rustup default 1.72.0

Output:

info: using existing install for '1.72.0-aarch64-unknown-linux-gnu'
info: default toolchain set to '1.72.0-aarch64-unknown-linux-gnu'

  1.72.0-aarch64-unknown-linux-gnu unchanged - rustc 1.72.0 (5680fa18f 2023-08-23)

Check:

rustc --version

Output:

rustc 1.72.0 (5680fa18f 2023-08-23)

OPTIONAL/NOTE: Manually installing and building RUST (takes a lot of time)

cd /tmp
wget https://static.rust-lang.org/dist/rust-1.72.0-aarch64-unknown-linux-gnu.tar.gz
tar -xzf rust-1.72.0-aarch64-unknown-linux-gnu.tar.gz
cd rust-1.72.0-aarch64-unknown-linux-gnu
sudo ./install.sh

Pytorch #

PyTorch is a Python package that provides two high-level features:

  • Tensor computation (like NumPy) with strong GPU acceleration
  • Deep neural networks built on a tape-based autograd system

You can reuse your favorite Python packages such as NumPy, SciPy, and Cython to extend PyTorch when needed.

There is a wheel installer (322MB) for pytoch, download it like this:

wget https://nvidia.box.com/shared/static/fjtbno0vpo676a25cgvuqc1wty0fkkg6.whl -O torch-1.10.0-cp36-cp36m-linux_aarch64.whl

Install the wheel:

sudo -H pip3 install torch-1.10.0-cp36-cp36m-linux_aarch64.whl

Output:

Processing ./torch-1.10.0-cp36-cp36m-linux_aarch64.whl
Collecting dataclasses; python_version < "3.7" (from torch==1.10.0)
  Downloading https://files.pythonhosted.org/packages/fe/ca/75fac5856ab5cfa51bbbcefa250182e50441074fdc3f803f6e76451fab43/dataclasses-0.8-py3-none-any.whl
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.6/dist-packages (from torch==1.10.0)
Installing collected packages: dataclasses, torch
Successfully installed dataclasses-0.8 torch-1.10.0

Remove the downloaded wheel installer

sudo rm torch-1.10.0-cp36-cp36m-linux_aarch64.whl

Check for pytorch modules in python:

python - <<'EOF'
import torch
print("Torch version:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())
print("GPU:", torch.cuda.get_device_name(0) if torch.cuda.is_available() else "-")
EOF

Expected output:

Torch version: 1.10.0
CUDA available: True
GPU name: NVIDIA Tegra X1

Prepare software with PIP3 – phase 2 #

Because this is an older device image we can only use the last supported versions on this hardware/OS, so install these modules with this command:

sudo -H pip3 install "Cython==0.29.36" "pyyaml==5.4.1" "numpy==1.19.5" "regex==2022.3.15" "accelerate==0.9.0"

Install diffusers with no dependancies, otherwise the install will fail:

sudo -H pip3 install --no-deps "diffusers==0.3.0"

//NOT WORKING Install safetensors (not as sudo, because RUST cannot be found as root) //NOT WORKING

pip3 install --user --no-cache-dir safetensors==0.2.8

Install Transformers with no dependancies, otherwise the install will fail:

sudo -H pip3 install --no-deps "transformers==4.18.0"

Tokenizers #

Download tokenizers with git:

git clone https://github.com/huggingface/tokenizers.git ~/tokenizers-src

Set version to 0.12.0 on this older hardware/OS

cd ~/tokenizers-src
git checkout v0.12.0

Output:

Note: checking out 'v0.12.0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 0eb7455f Preparing `0.12` release. (#967)

now we need to set base64ct to a specific version:

cd ~/tokenizers-src/tokenizers
cargo update -p base64ct --precise 1.7.3

Output:

    Updating crates.io index
    Updating crates.io index
 Downgrading base64ct v1.8.0 -> v1.7.3
note: pass `--verbose` to see 15 unchanged dependencies behind latest

Now let’s build, we need to set specific flags to ignore warnings otherwise the build will fail!:

cd ~/tokenizers-src/bindings/python
RUSTFLAGS="-A invalid_reference_casting" env "PATH=$HOME/.cargo/bin:$PATH" python3 setup.py build_rust --inplace

Test it with python on the commandline

python3 -c "import tokenizers; print(tokenizers.__version__)"

Expected output:

0.12.0

Remove source files:

cd ~/
sudo rm -rf tokenizers-src

Torchvision #

There is no ready-to-install version of torchvision for the Jetson NANO, so we have to build it.

Download torchvision 0.11.1 (1.22GB) that wil work with the hardware/OS with git:

cd ~/
git clone --branch v0.11.1 https://github.com/pytorch/vision.git