Table of Contents
Tor Exit Full Setup
- we currently use Debian Jessie; Ubuntu LTS is another good valid option
- git: we keep some scripts, configuration files and templates in a git repository. We clone the git repository to the local system and use symlinks when possible. For easier re-use in parts, the documentation below uses direct links to the configuration files at github.
SSH key authentication only
- ssh config with password auth and PAM disabled, no root login, x11forwarding disabled
- change port to some port of your chosing above 1024 (when using multiple IPs: bind to one IP only)
ssh-copy-id -p $SSH_PORT user@server
This config by defaults allows world access to the SSH port and Tor ports 80,443. Think about limiting that to an IP (range) you can connect from, or use portknocking. Also, this config is optimized for high bandwidth relays: in order to avoid the conntrack module, it allows all UDP in!
cd /etc wget https://raw.githubusercontent.com/torservers/server-config-templates/master/iptables.test.rules chmod 600 iptables.test.rules vi iptables.test.rules # update at least SSH port iptables-restore < iptables.test.rules
- test if you can still connect in another session :)
iptables-save > iptables.rules chmod 600 iptables.rules cd /etc/network/if-pre-up.d/ wget https://raw.githubusercontent.com/torservers/server-config-templates/master/iptables-restore chmod +x iptables-restore
some useful defaults
# configure hostname hostname yourservername.xyz vi /etc/hostname # also use yourservername.xyz vi /etc/hosts # update to yourservername.xyz yourservername # disable debian default that pulls in recommended packages: cd /etc/apt/apt.conf.d wget https://raw.githubusercontent.com/torservers/server-config-templates/master/06norecommends apt-get update && apt-get full-upgrade apt-get install sudo git less htop nload screen \ ntp apticron vnstat logcheck logcheck-database lsb-release apt-get remove --purge portmap sed -i -e 's/^# DIFF_ONLY/DIFF_ONLY/' /etc/apticron/apticron.conf # make apticron send diffs only vnstat -u -i eth0 # setup vnstat for correct interface sudoedit /etc/resolv.conf # remove search line cd /etc mv aliases aliases.dist wget https://raw.githubusercontent.com/torservers/server-config-templates/master/aliases sed -i 'email@example.comfirstname.lastname@example.org/' /etc/aliases newaliases
We upgrade from all available package sources, let it reboot if necessary, and send mail on errors. A reasonable configuration could be to limit upgrades to the security sources. (see comments in 50unattended-upgrades)
apt-get install unattended-upgrades wget -p -O /etc/apt/apt.conf.d/50unattended-upgrades https://raw.githubusercontent.com/torservers/server-config-templates/master/50unattended-upgrades cp /usr/share/unattended-upgrades/20auto-upgrades /etc/apt/apt.conf.d/20auto-upgrades # enable
# codename of current distribution DIST=`lsb_release -sc` # add tor sources echo "deb http://deb.torproject.org/torproject.org $DIST main" > /etc/apt/sources.list.d/torproject.list # only add experimental if you're on tor IRC/familiar with it echo "deb http://deb.torproject.org/torproject.org tor-experimental-0.2.8.x-$DIST main" >> /etc/apt/sources.list.d/torproject.list gpg --keyserver keys.gnupg.net --recv 886DDD89 # if down use eg. keyserver.ubuntu.com gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add - sudo apt-get update sudo apt-get install deb.torproject.org-keyring
apt-get install tor tor-geoipdb service tor stop cd /etc/tor rm * wget https://raw.githubusercontent.com/torservers/server-config-templates/master/torrc wget https://raw.githubusercontent.com/torservers/server-config-templates/master/tor-exit-notice.html # customize # for multi-process installations, change datadirectory and pidfile (see section further below)
Register at Tor Weather for basic monitoring.
Which Ports Should I Use?
We have simplified our setup and are currently not using a standalone webserver on our relay servers.
Generate Bandwidth Stats with vnstat(i)
You can generate local graphs and render them as images with vnstati. You could serve these images with a webserver. We don't use this any longer.
apt-get install vnstati # create empty files in root-owned /var/www and change owner to www-data cd /var/www touch vnstat.png vnstat_d.png vnstat_m.png vnstat.xml chown www-data:www-data vnstat*.* # set up cron job crontab -u www-data -e
*/10 * * * * /usr/bin/vnstati -vs -o /var/www/vnstat.png -i eth0 >/dev/null 2>&1 ;fi */10 * * * * /usr/bin/vnstati -d -o /var/www/vnstat_d.png -i eth0 >/dev/null 2>&1 ;fi 1 3 * * * /usr/bin/vnstati -m -o /var/www/vnstat_m.png -i eth0 >/dev/null 2>&1 ;fi 1 3 * * * /usr/bin/vnstat --xml > /var/www/vnstat.xml 2>/dev/null ;fi
Local DNS server
- we use unbound, a local caching DNS server
- more and more people report problems with unbound; you might want to use 'named' instead.
- for optimizations of unbound, see http://www.unbound.net/documentation/howto_optimise.html; we don't use any of them at the moment, it just works fine out of the box
- many people use Google DNS, and it is indeed among the better choices of free DNS. do not use openDNS or other DNS providers that filter DNS requests. for a discussion, see also Reminder: exit nodes probably shouldn't be using Google's DNS servers
apt-get install unbound vi /etc/resolv.conf # insert top: nameserver 127.0.0.1
Munin Resource Monitoring
Install munin-node and allow remote access from our webserver that runs munin to gather statistics at 22.214.171.124 ( https://www.torservers.net/munin/ ). munin-node is the “client side component”. You might also be interested in the munin "server side component" configuration.
apt-get install -y munin-node ln -s /usr/share/munin/plugins/netstat /etc/munin/plugins/netstat rm /etc/munin/plugins/http_loadtime rm /etc/munin/plugins/ntp_* rm /etc/munin/plugins/postfix_* rm /etc/munin/plugins/exim_* sed "s/allow \\^127\\\.0\\\.0\\\.1\\$/allow ^81\\\.7\\\.13\\\.16$/" -i /etc/munin/munin-node.conf /etc/init.d/munin-node restart
High Bandwidth Tweaks (>100 mbps?)
You might also be interested in this tor-relay thread regarding high speed relay tweaks: How to Run High Capacity Tor Relays
In general, as with all optimizations: you should only apply those that are necessary for you.
Multiple Tor Processes
Currently, Tor does not scale on multicore CPUs. One Tor process is able to handle around 100mbps of throughput. If your line supports more, you will need to run multiple Tor processes. There is a modified initscript that makes it easy to manage multiple Tor configurations on one machine.
The relevant tor*.cfg settings per relay are (change “0” to “1” etc):
DataDirectory /var/lib/tor/0 PidFile /var/run/tor/tor0.pid Log notice file /var/log/tor/notices0.log
Note that running more than two tor processes per IP address will result in those other nodes not being used on the network. You'll see the following message in your logs:
[notice] Heartbeat: It seems like we are not in the cached consensus.
## layout # ls /etc/tor tor0.cfg tor1.cfg tor2.cfg tor3.cfg
wget -O /etc/systemd/system/tor\@.service https://raw.githubusercontent.com/torservers/server-config-templates/master/tor@.service systemctl enable tor@0 systemctl enable tor@1 systemctl enable tor@2
system v init (pre jessie)
cd /etc/init.d wget -O tor https://gist.githubusercontent.com/7adietri/9122199/raw/4ed71b894eddbdfb0e241fa06bb583a19f0ccc89/tor chmod +x tor # /etc/init.d/tor start # starts tor 0-3 # /etc/init.d/tor stop # stops tor 0-3 # /etc/init.d tor reload tor2 tor3 # /etc/init.d/tor stop tor1
sysctl.conf kernel optimizations
cd /etc mv sysctl.conf sysctl.conf.dist wget https://raw.githubusercontent.com/torservers/server-config-templates/master/sysctl.conf # go through the settings once again! some only useful with large memory and CPU # better tweaking probably possible; magic involved sysctl -p
edit rc.local and add sysctl -p
Set MaxBandwidth to line maximum, eg. for GBit:
sed "s/MaxBandwidth 100/MaxBandwidth 1000/g" -i /etc/vnstat.conf /etc/init.d/vnstat restart # don't reload; will stop vnstat from updating its db...
Might be useful in some cases. Only optimize when you need to!
# remove "exit" from rc.local, then echo 'ifconfig eth0 txqueuelen 20000' >> /etc/rc.local # Play with it. For GBit I've found values between 8000 and 16000 to be very useful, but it seems to be hardware dependent
Receive Packet Steering (RPS)
This is highly unlikely to be necessary on modern kernels! If one CPU core is used much more than others (check eg. with htop), enable RPS on kernels >= 2.6.35.
auto eth1 iface eth1 inet static address 123.456.789.01 netmask 255.255.255.255 gateway 012.345.678.90 up echo f > /sys/class/net/eth1/queues/rx-0/rps_cpus
Receive Flow Steering (RFS)
This is highly unlikely to be necessary on modern kernels! Receive Flow Steering (RFS), also introduced with kernel 2.6.35, might help if you run multiple Tor processes, but test first and monitor CPU usage across the CPU cores.
echo 16384 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt echo 16384 > /proc/sys/net/core/rps_sock_flow_entries
To make it permanent put it some place useful (can it be put into sysctl.conf?)
AES-NI Crypto Acceleration
Recent Intel CPUs and upcoming AMDs support a native AES crypto acceleration extension called AES-NI. It is well worth enabling and will save a lot of CPU cycles.
Many motherboards ship with AES-NI disabled. You can check if it is enabled:
# cat /proc/cpuinfo | grep aes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 x2apic popcnt aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid
Enable AES-NI in Tor
OpenSSL 1.0.1 does not come with an extra module and should directly support AES-NI. You should not use older versions of OpenSSL.
- monitoring: Zabbix?
- disable webserver referer and user-agent logging, too?
- firewall: is tarpitting really useful for us?