For the server (the nice grey cabinet to the left) to reach all those systems at the same time, a VLAN setup was required, so we set up a VLAN trunk to a nearby switch. Then the different appliances were connected to different un-tagged switch ports in dedicated VLANs – the first appliance was connected to a switch port in VLAN 10, the next to a switch port in VLAN 11, etc. Corresponding VLAN interfaces were configured on the server, and a local IP address was configured on each. To keep a minimum level of sanity, I configured the VLAN interface on vlan10 as 192.168.1.10, while vlan11 got 192.168.1.11, etc. To avoid too much confusion on the switch, I also set the MAC address on each VLAN interface to distinct values.

Next step was to configure correct IP routing using iproute2:
# VLAN 10
/sbin/ip link add link eth1 name eth1.10 type vlan id 10
/sbin/ip link set dev eth1.10 down
/sbin/ip link set dev eth1.10 address 00:04:de:ad:be:10
/sbin/ip link set dev eth1.10 up
/sbin/ip addr add local 192.168.1.10/24 brd 192.168.1.255 dev eth1.10
/sbin/ip route add 192.168.1.0/24 dev eth1.10 proto kernel scope link src 192.168.1.10 table 10
/sbin/ip rule add from 192.168.1.10 lookup 10 prio 1010
# VLAN 11
/sbin/ip link add link eth1 name eth1.11 type vlan id 11
/sbin/ip link set dev eth1.11 down
/sbin/ip link set dev eth1.11 address 00:04:de:ad:be:11
/sbin/ip link set dev eth1.11 up
/sbin/ip addr add local 192.168.1.11/24 brd 192.168.1.255 dev eth1.11
/sbin/ip route add 192.168.1.0/24 dev eth1.11 proto kernel scope link src 192.168.1.11 table 11
/sbin/ip rule add from 192.168.1.11 lookup 11 prio 1011

At this point I could reach the different appliances from the different source IPs:
# ping -I 192.168.1.10 192.168.1.1 -c1
PING 192.168.1.1 (192.168.1.1) from 192.168.1.10 : 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.119 ms

Snooping with tcpdump on the different VLAN interfaces, also showing the MAC addresses, confirmed that the pings were reaching the correct appliance.

Then I set up corresponding IP addresses on the outside interface (eth0). The 10.0.0.195 address matches the device in VLAN 10, 10.0.0.196 matches VLAN 11 and so on. The outside IP addresses are official IP addresses, but I’ve changed them to protect the innocent.

/sbin/ip addr add local 10.0.0.195/24 dev eth0
/sbin/ip rule add to 10.0.0.195 fwmark 10 table 10
/sbin/ip rule add fwmark 10 lookup 10
/sbin/ip addr add local 10.0.0.196/24 dev eth0
/sbin/ip rule add to 10.0.0.196 fwmark 11 table 11
/sbin/ip rule add fwmark 11 lookup 11

To make sure the packets were being identified correctly all the way through, they were tagged upon reaching the outside interface, based on the IP on which they arrived.

So far so good, now it was time for iptables to show its powers:
# VLAN 10
/sbin/iptables -t mangle -A PREROUTING -i eth0 -d 10.0.0.195 -j MARK --set-mark 10
/sbin/iptables -t nat -A PREROUTING -i eth0 -d 10.0.0.195 -j DNAT --to-destination 192.168.1.1
/sbin/iptables -A FORWARD -m mark --mark 10 -j ACCEPT
/sbin/iptables -t nat -A POSTROUTING -m mark --mark 10 -j SNAT --to-source 192.168.1.10
# VLAN 11
/sbin/iptables -t mangle -A PREROUTING -i eth0 -d 10.0.0.196 -j MARK --set-mark 11
/sbin/iptables -t nat -A PREROUTING -i eth0 -d 10.0.0.196 -j DNAT --to-destination 192.168.1.1
/sbin/iptables -A FORWARD -m mark --mark 11 -j ACCEPT
/sbin/iptables -t nat -A POSTROUTING -m mark --mark 11 -j SNAT --to-source 192.168.1.11

The above commands maps the outside addresses on eth0 to the corresponding VLANs on the inside. Note that every incoming packet is destination NATed (DNAT) to 192.168.1.1 while they are being source NATed (SNAT) based on the VLAN to which they are mapped. As the appliances were not able to route traffic, all communication had to look like it happened on the local network only.

Now I expected to be able to access each appliance from the outside, but it turned out that only one appliance could be reached. tcpdumping on each VLAN interface confirmed that traffic from the outside reached each appliance, and the appliance responded, but the response never came all the way back. This was rather puzzling as nothing appeared in any log. However, my colleagues reminded me about rp_filter, which dropped weird traffic without logging anything. Thus, the key that unlocked everything was this:
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth1.10/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth1.11/rp_filter

And voila – from the outside world, we were able to reach every single appliance on the inside, all with the same unroutable RFC1918 address.

tcpdump -n -s0 -i ppp0 'icmp[icmptype] != icmp-echo and icmp[icmptype] != icmp-echoreply'

Update: Of course, this exact syntax is documented on tcpdump’s own site…

This is how to do the same with Varnish. The example suggests that you forbid hotlinking to anything under http://www.example.com/fun/. The code will of course need to be added to whatever else exists in vcl_recv.

sub vcl_recv {
  if (req.http.host == "www.example.com" &&
  req.url ~ "^/fun/" &&
  (req.http.referer && req.http.referer !~ "^http://www.example.com/")) {
    error 403 "No hotlinking please";
  }
}

Having enabled it through its IE-specific web UI, I logged in to the (rather poorly documented) CLI afterwards to check what had changed in the config. And the single, magic configuration line that enabled spanning-tree on the switch was this one:

spanning-tree

*sigh*

The files are split into Location and Blocks, so we create two tables to accommodate this:

CREATE TABLE location (
 LocId INT(8) NOT NULL,
 PRIMARY KEY (LocId),
 country CHAR(2),
 region VARCHAR(255),
 city VARCHAR(255),
 postalCode VARCHAR(255),
 latitude DECIMAL(20,17),
 longitude DECIMAL(20,17),
 metroCode VARCHAR(50),
 areaCode VARCHAR(50)
) ENGINE=INNODB;
CREATE TABLE blocks (
 startIpNum INT(10) NOT NULL,
 EndIpNum INT(10) NOT NULL,
 LocId INT(8) NOT NULL,
 FOREIGN KEY (LocID) REFERENCES location(LocId) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=INNODB;

Now it’s time to import the data from the files. Since LocId is a foreign key to the blocks table, location must be inserted first.

mysql> LOAD DATA LOCAL INFILE '/root/GeoLiteCity_20090601/GeoLiteCity-Location.csv' INTO TABLE location FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' (LocId, country, region, city, postalCode, latitude, longitude, metroCode, areaCode);
Query OK, 248276 rows affected, 13 warnings (9.65 sec)
Records: 248277 Deleted: 0 Skipped: 1 Warnings: 9
mysql> LOAD DATA LOCAL INFILE '/root/GeoLiteCity_20090601/GeoLiteCity-Blocks.csv' INTO TABLE blocks FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' (startIpNum, EndIpNum, LocId);
Query OK, 4132041 rows affected, 65535 warnings (15 min 20.44 sec)
Records: 4132041 Deleted: 0 Skipped: 0 Warnings: 2739156

That done, we try a select.

mysql> select A.country from location A
  JOIN blocks B on A.LocId = B.LocId
  AND 404232216 >= startIpNum AND 404232216 <= EndIpNum;
+---------+
| country |
+---------+
| US      |
+---------+
1 row in set (2 min 3.58 sec)

As I had no indexes yet, the query took some time. So let’s add them:

mysql> CREATE INDEX startIp_ind ON blocks (startIpNum);
Query OK, 4132041 rows affected (9 min 22.23 sec)
Records: 4132041 Duplicates: 0 Warnings: 0
mysql> CREATE INDEX endIp_ind ON blocks (endIpNum);
Query OK, 4132041 rows affected (9 min 22.33 sec)
Records: 4132041 Duplicates: 0 Warnings: 0
mysql> CREATE INDEX LocId_ind ON location (LocId);
Query OK, 248276 rows affected (4.08 sec)
Records: 248276 Duplicates: 0 Warnings: 0

And voila:

mysql> select A.country from location A
  JOIN blocks B on A.LocId = B.LocId
  AND 404232216 >= startIpNum AND 404232216 <= EndIpNum;
+---------+
| country |
+---------+
| US      |
+---------+
1 row in set (0.55 sec)

Also, a better SQL query could’ve been more efficient, but hey – this is for illustrative purposes.

Er hjemmesiden sikker og privat?

Ja. Denne hjemmesiden blir operert av den Amerikanske myndigheten, og det benyttes teknologi som gjør at ikke autorisert personer får tilgang til informasjonen du legger inn og ser på . I tillegg opererer denne hjemmesiden ut i fra de lover og regler som den Amerikanske Stats lov om personvern. Denne Personvernsbekreftelsen forsikrer deg om at ditt personvern vil bli ivaretattinformation.

Så da burde jo alt være i orden.

Oppdatering – Kollega Knut viste meg popup-en han fikk da han gikk inn på de samme sidene. Med de forutsetningene burde de fleste muligheter være dekket.

log_ip 10.20.30.40

Reboot your unit and watch the logs roll in. Verify by checking what syslog is up to:

root@OpenWRT:~# ps -ef | grep syslog
87 root 1928 S syslogd -C16 -L -R 10.20.30.40

Now, that said, I also found how to set the name servers (for /etc/resolv.conf) the correct way – adding an option entry under a suitable interface definition in /etc/config/network. Note that for several DNS servers the IPs should be space separated, like this:

option ‘dns’ ’10.0.0.1 10.0.0.2′

This creates an /etc/resolv.conf looking like this one:

nameserver 10.0.0.1
nameserver 10.0.0.2

I got great help from this article on tweaking VLANs on the Xen host, and this article on VLANs with kvm. Thanks!