Setup Entware-ng-3x on Asuswrt-merlin

Entware-ng-3x is a modern alternative to Optware.

For those unfamiliar with Optware: it's a software repository that offers various software programs that can be installed on your router. They allow you to add new functionality to your router (provided you have the know-how to properly configure them).

Entware-ng-3x system library is specially modified (patched) so that standard linux files that are normally located in /etc directory are now located in /opt/etc/ directory. To simplify things let's consider /etc/passwd file. On Asuswrt-merlin it normally looks something like:

admin:x:0:0:admin:/root:/bin/sh
nas:x:100:100:nas:/dev/null:/dev/null
nobody:x:65534:65534:nobody:/dev/null:/dev/null
tor:x:65533:65533:tor:/dev/null:/dev/null

We call standard Entware-ng-3x installation the case, when /opt/etc/passwd (shadow, group, gshadow, shells) files are symbolic links to the equivalent files in /etc directory. When Entware-ng binaries need to access /opt/etc/passwd file they access /etc/passwd file. This is equivalent of Entware-ng behavior.

We call alternative Entware-ng-3x installation the case when /opt/etc/passwd and /opt/etc/group are real files (not symlinks). Entware-ng-3x binaries use user names, group names, passwords from these files. They can be different from Asuswrt-merlin firmware. The original Asuswrt-merlin firmware has admin as a superuser, Entware-ng-3x always has root as a superuser.

Important: Please remove Optware, Entware-ng, Entware.arm if it is currently installed.

Uninstall Optware (optional)

You cannot use both Optware and Entware-ng-3x at the same time. Entware-ng-3x cannot be used simultaneously with Optware.

Important: Asus's Download Master is based on Optware, and therefore is NOT compatible with Entware-ng-3x. You will have to uninstall Download Master.

After uninstalling, you should make sure asusware.arm or asusware. directory on mounted disk partition is deleted. Otherwise, Entware-ng-3x won't work properly.

Uninstall Entware (optional)

You cannot use both Entware-ng and Entware-ng-3x at the same time. Entware-ng-3x cannot be used simultaneously with Entware-ng.

SSH to your router, change 192.168.1.1 with the ip address of your router

$ ssh admin@192.168.1.1

Delete /tmp/opt symlink

$ rm /tmp/opt

Delete start scripts

$ rm /jffs/scripts/services-start
$ rm /jffs/scripts/services-stop
$ rm /jffs/scripts/post-mount

Disable custom scripts and configs from /jffs

$ nvram set jffs2_scripts=0
$ nvram commit

Reboot router

$ reboot

Unmount and format the USB drive

$ df -h
$ umount -l /dev/sda1
$ mkfs.ext2 /dev/sda1

Reboot router

$ reboot

Prerequisites

Important: Please check kernel version of your router using uname -a command. Kernel version should be not less then 3.2.40!

$ uname -a
Linux RT-AC86U-E5F0 4.1.27 #2 SMP PREEMPT Sat Dec 2 01:19:36 EST 2017 aarch64 ASUSWRT-Merlin

The goal is to mount /opt folder to some ext2|ext3|ext4 partition on a USB drive, which will be automatically mounted on every boot, so make sure to plug in a USB drive before installation.

In this example the target is an Asus RT-AC86U. The RT-AC86U contains a Broadcom BCM4906 SoC. It integrates two Cortex-A53 ARMv8-A cores that Broadcom has customized. Armv8-A has two execution states AArch32 and AArch64. AArch32 provides backwards compatibility with ARMv7.

This all means you can use a ready made installation scripts for ARMv7.

SSH to your router, change 192.168.1.1 with the ip address of your router

$ ssh admin@192.168.1.1

Change to /tmp/share

$ cd /tmp/share

Download entware-setup.sh from the new Asuswrt-merlin development branch

$ wget https://raw.githubusercontent.com/RMerl/asuswrt-merlin.382/master/release/src/router/others/entware-setup.sh

For standard installation (armv7,std)

$ sed -i 's~http://pkg.entware.net/binaries/armv7/installer/entware_install.sh~http://entware-3x.zyxmon.org/binaries/armv7/installer/install_std.sh~g' /tmp/share/entware-setup.sh

For alternative installation (armv7,alt)

$ sed -i 's~http://pkg.entware.net/binaries/armv7/installer/entware_install.sh~http://entware-3x.zyxmon.org/binaries/armv7/installer/install_alt.sh~g' /tmp/share/entware-setup.sh

Correct permissions to /tmp/share/entware-setup.sh

$ chmod +x /tmp/share/entware-setup.sh

Unmount and format the USB drive

$ df -h
$ umount -l /dev/sda1
$ mkfs.ext2 /dev/sda1

Install Entware-ng-3x

$ /tmp/share/entware-setup.sh

admin@RT-AC86U-E5F0:/tmp/share# /tmp/share/entware-setup.sh
 Info:  This script will guide you through the Entware installation.
 Info:  Script modifies "entware" folder only on the chosen drive,
 Info:  no other data will be changed. Existing installation will be
 Info:  replaced with this one. Also some start scripts will be installed,
 Info:  the old ones will be saved on Entware partition with name
 Info:  like /tmp/mnt/sda1/jffs_scripts_backup.tgz

 Info:  Looking for available partitions...
[1] --> /tmp/mnt/sda1
 =>  Please enter partition number or 0 to exit

Choose a partition where Entware-ng-3x should be installed, in this case [1] --> /tmp/mnt/sda1

[0-1]: 1
 Info:  /tmp/mnt/sda1 selected.

Output on a clean install should look like this

Info:  Creating /tmp/mnt/sda1/entware folder...
Info:  Creating /tmp/opt symlink...
Info:  Creating /jffs scripts backup...
tar: /jffs/scripts/*: No such file or directory
tar: error exit delayed from previous errors
Info:  Modifying start scripts...
Info:  Enabling custom scripts and configs from /jffs...

Output on a successful install should look like this

Info: Congratulations!
Info: If there are no errors above then Entware-3x was successfully initialized.
Info: Add /opt/bin & /opt/sbin to your PATH variable
Info: Add '/opt/etc/init.d/rc.unslung start' to firmware startup script for Entware-3x services to start

The script has created a new directory named entware

admin@RT-AC86U-E5F0:/tmp/share# cd /opt
admin@RT-AC86U-E5F0:/tmp/mnt/sda1/entware#

Entware-ng-3x installs its own busybox copy. The PATH variable has /opt/bin and /opt/sbin before /bin and /sbin

$ which busybox
/opt/bin/busybox

Setup (alternative installation)

If this is an alternative Entware-3x installation, it is recommended to install and setup Entware-ng-3x ssh server and use it instead of a firmware supplied one. There are two ssh servers in Entware-ng-3x - dropbear and openssh. You can install dropbear or openssh as an ssh server.

The superuser root has password 12345 that can be changed after the first login and is independent from Asuswrt-merlin firmware.

Install dropbear

$ opkg update
$ opkg install dropbear

You need to edit /opt/etc/init.d/S51dropbear startup script and change the line PORT=22 to PORT=2222 to make dropbear ssh port different from the system's one.

$ sed -i 's~PORT=22~PORT=2222~g' /opt/etc/init.d/S51dropbear

Run dropbear

$ /opt/etc/init.d/S51dropbear start

Check if dropbear is currently running

$ /opt/etc/init.d/S51dropbear status
dropbear already running

SSH to your router, change 192.168.1.1 with the ip address of your router

$ ssh root@192.168.1.1 -p 2222

References

1. Entware-ng-3x – General installation notes
2. Entware-ng-3x – Install on QNAP NAS
3. Asuswrt-merlin – Entware
4. Asuswrt-merlin – USB Storage setup

How to compile a package for Entware-ng

For the case where Entware-ng doesn't have the package on board that you wish to use, you have the option to compile an Entware-ng package yourself.

Prerequisites

Meet OpenWrt dependencies

First setup an environment that meets OpenWrt buildroot dependencies as can be read in OpenWrt build system – Installation. This is because Entware-ng is a modified OpenWrt environment.

Setup Entware-ng-3x on macOS

Install Xcode command line tools or Xcode from the App Store

$ xcode-select --install

Install Homebrew

$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ echo 'export PATH="/usr/local/sbin:$PATH"' >> ~/.bash_profile

Add duplicates repository to homebrew for grep formulae:

$ brew tap homebrew/dupes

Install additional formulae:

$ brew install findutils
$ brew install gawk
$ brew install gnu-getopt
$ brew install gnu-tar
$ brew install grep
$ brew install quilt
$ brew install wget

To get rid of "date illegal option" you can add to your .bash_profile:

$ echo 'export PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"' >> ~/.bash_profile

gnu-getopt is keg-only, so force linking it:

$ brew link gnu-getopt --force

OS X by default comes with a case-insensitive filesystem. OpenWrt Buildroot won't build on that. As a workaround, create a (Sparse) case-sensitive disk-image that you then mount in Finder and use as build directory:

$ hdiutil create -size 10g -type SPARSE -fs "Case-sensitive HFS+" -volname Entware-ng Entware-ng-3x.sparseimage
$ hdiutil attach Entware-ng-3x.sparseimage

Change to '/Volumes/Entware-ng'

$ cd /Volumes/Entware-ng

Then setup Entware-ng-3x by cloning its repository:

$ git init
$ git remote add origin https://github.com/Entware-for-kernel-3x/Entware-ng-3x.git
$ git fetch
$ git checkout -t origin/master

Update entware-packages-3x feed

$ ./scripts/feeds update packages
$ ./scripts/feeds install -a -p packages

Setup the configuration for the target device

In this example the target is an Asus RT-AC86U. The RT-AC86U contains a Broadcom BCM4906 SoC. It integrates two Cortex-A53 ARMv8-A cores that Broadcom has customized. Armv8-A has two execution states AArch32 and AArch64. AArch32 provides backwards compatibility with ARMv7.

This all means you can use a ready made configuration file that is located in the 'configs' directory with the name armv7-3x.config:

$ cp configs/armv7-3x.config .config

Prepare build environement

The full execution on OpenWrt Buildroot a.k.a. rebuild repo a.k.a. make is not necessary. You only need:

$ make tools/install
$ make toolchain/install

You can speed up compilation on multiprocessor systems by running several build threads simultaneously by adding -jN:

$ make -j4 tools/install
$ make -j4 toolchain/install

Error in tools or toolchain

In case there is an error in the tools/install or toolchain/install phase, and you change your configuration, remember to run a make dirclean instead of make clean.

Compile squid package

Now you should be ready to compile an individual package. For starters you can test make package/.../compile using a tiny package, for example squid:

$ make -j4 package/squid/compile

If something goes wrong, turn on verbose mode by adding V=s:

$ make -j4 package/squid/compile V=s

When everything succeeds your package ends up in a subdirectory of the Entware-ng 'bin' directory. For this armv7-3x based example, you should find the .ipk file at 'bin/targets/armv7-3x/generic-glibc/packages':

$ ls bin/targets/armv7-3x/generic-glibc/packages | grep squid
squid-mod-cachemgr_3.5.27-1_armv7-3x.ipk
squid_3.5.27-1_armv7-3x.ipk

When you see the ...armv7-3x.ipk output, your first package is successfully build.

Unlink 'gnu-getopt' on macOS

$ brew unlink gnu-getopt

Install compiled squid package

Change to the directory where you have the squid_3.5.27-1_armv7-3x.ipk file

$ cd bin/targets/armv7-3x/generic-glibc/packages

Copy the .ipk file to '/tmp/share' directory, change 192.168.1.1 with the ip address of your router

$ scp squid_3.5.27-1_armv7-3x.ipk admin@192.168.1.1:/tmp/share

SSH to your router, change 192.168.1.1 with the ip address of your router

$ ssh admin@192.168.1.1
$ opkg remove squid (if previously installed)
$ opkg install /tmp/share/squid_3.5.27-1_armv7-3x.ipk
$ opkg info squid

The output from the squid -v should start with something like:

Squid Cache: Version 3.5.27

Correct permissions to '/dev/shm' directory to fix error: squid: Ipc::Mem::Segment::create failed to shm_open(/squid-cf__metadata.shm): (13) Permission denied

$ sed -i '/PREARGS=""/a PRECMD="chmod 777 /dev/shm"' /opt/etc/init.d/S22squid

Run squid

$ /opt/etc/init.d/S22squid start
Starting squid...              done.

Check if squid is currently running

$ /opt/etc/init.d/S22squid check
Checking squid...              alive.

Run startup script (optional)

You might want to start your package automatically when the router starts. For this purpose the router does execute scripts located in the directory '/opt/etc/init.d/'.

Entware-ng init system

Entware uses an 'init.d' directory (where init is short for initialization) containing scripts. Entware uses "rc.func" as a wrapper to provide its main and default functionality. Entware also uses "rc.unslung" as the helper script to start and stop all '/opt/etc/init.d/' scripts. To be more precise, rc.unslung will process all executable files that start with a capital S and will do that in sorted order. Except for stopping, restarting and killing, then the executable S* init script files will be processed in reverse sorted order.

rc.func

Learn yourself bash scripting and have a look inside "rc.func" to understand its functionality. By this rc.func template, the available commands for any init script are as follows:

start         Start the service
stop          Stop the service
restart       Restart the service
check         Check the service (dead or alive)
kill          Kill the service
reconfigure   Reload configuration files (sends SIGHUP signal)

All these arguments can be passed to the script when run. For example, to stop squid call it with stop:

# /opt/etc/init.d/S22squid stop

Entware-ng init script

An example Entware init script, S22squid from the net package, looks like this:

#!/bin/sh

ENABLED=yes
PROCS=squid
ARGS=""
PREARGS=""
DESC=$PROCS
PATH=/opt/sbin:/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

. /opt/etc/init.d/rc.func

References

1. Entware-ng – Compile packages from sources
2. Entware-ng – How to add a new package
3. OpenWrt Buildroot – Installation
4. OpenWrt Buildroot – Installation on macOS
5. OpenWrt Buildroot – Feeds
6. OpenWrt Buildroot – How to Build a Single Package
7. OpenWrt Buildroot – Usage

Useful links

1. Entware-ng-3x Packages list
2. Entware Packages list
3. Squid Versions

ssh

iptables

#iptables -A PREROUTING -t mangle -p tcp --dport 80 -s 192.168.1.1 ! -d `nvram get lan_ipaddr`/`nvram get lan_netmask` -j ACCEPT
#iptables -A PREROUTING -t mangle -p tcp --dport 80 -j MARK --set-mark 3
#ip rule add fwmark 3 table 2
#ip route add default via 192.168.1.1 dev br0 table 2
#iptables -A PREROUTING -t mangle -p tcp --dport 80 -s 192.168.1.xxx -j ACCEPT
#iptables -A PREROUTING -t nat -i eth0 -p --dport 80 -j REDIRECT --to-port 3128

squid

opkg install squid
sed -i '/PREARGS=""/a PRECMD="chmod 777 /dev/shm"' /opt/etc/init.d/S22squid
nano /opt/etc/squid/squid.conf
squid -k parse
/opt/etc/init.d/S22squid start
/opt/etc/init.d/S22squid check

config

mkdir /opt/var/log/squid/
nano /opt/etc/squid/squid.conf
squid -k reconfigure

commands

#squid -k rotate
#squid -k reconfigure
#squid -k kill

log

tail -f /opt/var/log/squid/access.log | grep -i 192.168.1.xxx

nvram

nvram get wan_ipaddr
nvram get lan_ipaddr
nvram get lan_netmask

opkg

opkg update
opkg list_installed
opkg list-upgradable

debug

cat /etc/resolv.conf
free -m
time nslookup -debug -d2 8.8.8.8
watch 'dmesg | tail -20'
ls -la /tmp/mnt
netstat -tuln
rm /opt/var/run/squid.pid

uninstall

opkg status squid
opkg remove --autoremove squid

performance

opkg install procps-ng-top
top -p `pgrep -d, -f squid`
top -p `pgrep -d, -f program1`, `pgrep -d, -f program2` (upper case P) makes top order by CPU