Discussion:
[quagga-users 9514] Zebra not installing default route into kernel
Jared Gillis
17 years ago
Permalink
Hello all,

I'm doing some lab work with zebra on a linux box and one of the goals is to have the linux server learn two default routes from upstream (directly connected) routers and failover its default route to whichever of the two BGP sessions that are up (no preference on which if both are up).
I've got the sessions up and I'm learning default from both, but when I drop the session to which default is currently pointing, the kernel routing table is not updated with a new default.

Here's relevant info:
bgpd.conf:

router bgp 64512
neighbor foo peer-group
neighbor foo remote-as 64512
neighbor foo soft-reconfiguration inbound
neighbor 208.106.121.49 peer-group foo
neighbor 208.106.121.53 peer-group foo

bgpd show ip bgp:
# show ip bgp
BGP table version is 0, local router ID is 208.106.123.141
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
*>i0.0.0.0 208.106.123.97 100 0 7065 i
* i 208.106.123.105 100 0 7065 i
*>i208.106.121.48/30
208.106.121.49 10 100 0 ?
*>i208.106.121.52/30
208.106.121.53 10 100 0 ?
*>i208.106.123.96/30
208.106.121.49 10 100 0 ?
*>i208.106.123.100/30
208.106.121.49 10 100 0 ?
*>i208.106.123.104/30
208.106.121.53 10 100 0 ?
*>i208.106.123.108/30
208.106.121.53 10 100 0 ?
*>i208.106.123.138/32
208.106.121.49 10 100 0 ?
*>i208.106.123.139/32
208.106.121.53 10 100 0 ?
*> 208.106.123.141/32
0.0.0.0 0 32768 i

Total number of prefixes 10

zebra show ip route:
# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
I - ISIS, B - BGP, > - selected route, * - FIB route

B 0.0.0.0/0 [200/0] via 208.106.123.97 inactive, 00:01:15
K>* 0.0.0.0/0 via 208.106.121.49, eth0
C>* 127.0.0.0/8 is directly connected, lo
K>* 169.254.0.0/16 is directly connected, eth0
B 208.106.121.48/30 [200/10] via 208.106.121.49 inactive, 00:01:15
C>* 208.106.121.48/30 is directly connected, eth0
B 208.106.121.52/30 [200/10] via 208.106.121.53 inactive, 00:01:20
C>* 208.106.121.52/30 is directly connected, eth1
B>* 208.106.123.96/30 [200/10] via 208.106.121.49, eth0, 00:01:15
B>* 208.106.123.100/30 [200/10] via 208.106.121.49, eth0, 00:01:15
B>* 208.106.123.104/30 [200/10] via 208.106.121.53, eth1, 00:01:20
B>* 208.106.123.108/30 [200/10] via 208.106.121.53, eth1, 00:01:20
B>* 208.106.123.138/32 [200/10] via 208.106.121.49, eth0, 00:01:15
B>* 208.106.123.139/32 [200/10] via 208.106.121.53, eth1, 00:01:20
C>* 208.106.123.141/32 is directly connected, lo

kernel routing table:
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
208.106.123.139 208.106.121.53 255.255.255.255 UGH 10 0 0 eth1
208.106.123.138 208.106.121.49 255.255.255.255 UGH 10 0 0 eth0
208.106.123.108 208.106.121.53 255.255.255.252 UG 10 0 0 eth1
208.106.123.104 208.106.121.53 255.255.255.252 UG 10 0 0 eth1
208.106.123.100 208.106.121.49 255.255.255.252 UG 10 0 0 eth0
208.106.121.52 0.0.0.0 255.255.255.252 U 0 0 0 eth1
208.106.123.96 208.106.121.49 255.255.255.252 UG 10 0 0 eth0
208.106.121.48 0.0.0.0 255.255.255.252 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 208.106.121.49 0.0.0.0 UG 0 0 0 eth0

# ip route sh
208.106.123.139 via 208.106.121.53 dev eth1 proto zebra metric 10 equalize
208.106.123.138 via 208.106.121.49 dev eth0 proto zebra metric 10 equalize
208.106.123.108/30 via 208.106.121.53 dev eth1 proto zebra metric 10 equalize
208.106.123.104/30 via 208.106.121.53 dev eth1 proto zebra metric 10 equalize
208.106.123.100/30 via 208.106.121.49 dev eth0 proto zebra metric 10 equalize
208.106.121.52/30 dev eth1 proto kernel scope link src 208.106.121.54
208.106.123.96/30 via 208.106.121.49 dev eth0 proto zebra metric 10 equalize
208.106.121.48/30 dev eth0 proto kernel scope link src 208.106.121.50
169.254.0.0/16 dev eth0 scope link
default via 208.106.121.49 dev eth0

When I ifdown'ed eth0, the eth0 routes were removed properly, but a new default route was not added towards the neighbor on eth1.
I'd appreciate any thoughts or suggestions.
Thanks!
--
Jared Gillis - ***@corp.sonic.net Sonic.net, Inc.
Network Operations 2260 Apollo Way
707.522.1000 (Voice) Santa Rosa, CA 95407
707.547.3400 (Support) http://www.sonic.net/
Jeremy Jackson
17 years ago
Permalink
Post by Jared Gillis
# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
I - ISIS, B - BGP, > - selected route, * - FIB route
B 0.0.0.0/0 [200/0] via 208.106.123.97 inactive, 00:01:15
K>* 0.0.0.0/0 via 208.106.121.49, eth0 <----- HERE IT IS
Maybe you can try removing that and restarting zebra/bgpd.
...
--
Jeremy Jackson
Coplanar Networks
(519)489-4903
http://www.coplanar.net
***@coplanar.net
Jared Gillis
17 years ago
Permalink
Hmm, I just tried that and no luck. After a 'route del default', there's no kernel route.
I restarted zebra and bgpd and while bgpd is receiving a default route, zebra's not putting anything into the kernel route table:

# show ip bgp
BGP table version is 0, local router ID is 208.106.123.141
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
* i0.0.0.0 208.106.123.97 100 0 7065 i
* i 208.106.123.105 100 0 7065 i
*>i208.106.121.48/30
208.106.121.49 10 100 0 ?
*>i208.106.121.52/30
208.106.121.53 10 100 0 ?
*>i208.106.123.96/30
208.106.121.49 10 100 0 ?
*>i208.106.123.100/30
208.106.121.49 10 100 0 ?
*>i208.106.123.104/30
208.106.121.53 10 100 0 ?
*>i208.106.123.108/30
208.106.121.53 10 100 0 ?
*>i208.106.123.138/32
208.106.121.49 10 100 0 ?
*>i208.106.123.139/32
208.106.121.53 10 100 0 ?
*> 208.106.123.141/32
0.0.0.0 0 32768 i

# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
I - ISIS, B - BGP, > - selected route, * - FIB route

C>* 127.0.0.0/8 is directly connected, lo
K>* 169.254.0.0/16 is directly connected, eth0
B 208.106.121.48/30 [200/10] via 208.106.121.49 inactive, 00:01:45
C>* 208.106.121.48/30 is directly connected, eth0
B 208.106.121.52/30 [200/10] via 208.106.121.53 inactive, 00:01:49
C>* 208.106.121.52/30 is directly connected, eth1
B>* 208.106.123.96/30 [200/10] via 208.106.121.49, eth0, 00:01:45
B>* 208.106.123.100/30 [200/10] via 208.106.121.49, eth0, 00:01:45
B>* 208.106.123.104/30 [200/10] via 208.106.121.53, eth1, 00:01:49
B>* 208.106.123.108/30 [200/10] via 208.106.121.53, eth1, 00:01:49
B>* 208.106.123.138/32 [200/10] via 208.106.121.49, eth0, 00:01:45
B>* 208.106.123.139/32 [200/10] via 208.106.121.53, eth1, 00:01:49
C>* 208.106.123.141/32 is directly connected, lo

# ip route sh
208.106.123.139 via 208.106.121.53 dev eth1 proto zebra metric 10 equalize
208.106.123.138 via 208.106.121.49 dev eth0 proto zebra metric 10 equalize
208.106.123.108/30 via 208.106.121.53 dev eth1 proto zebra metric 10 equalize
208.106.123.104/30 via 208.106.121.53 dev eth1 proto zebra metric 10 equalize
208.106.123.100/30 via 208.106.121.49 dev eth0 proto zebra metric 10 equalize
208.106.121.52/30 dev eth1 proto kernel scope link src 208.106.121.54
208.106.123.96/30 via 208.106.121.49 dev eth0 proto zebra metric 10 equalize
208.106.121.48/30 dev eth0 proto kernel scope link src 208.106.121.50
169.254.0.0/16 dev eth0 scope link
Post by Jeremy Jackson
Post by Jared Gillis
# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
I - ISIS, B - BGP, > - selected route, * - FIB route
B 0.0.0.0/0 [200/0] via 208.106.123.97 inactive, 00:01:15
K>* 0.0.0.0/0 via 208.106.121.49, eth0 <----- HERE IT IS
Maybe you can try removing that and restarting zebra/bgpd.
Jeremy Jackson
17 years ago
Permalink
Are both of the next-hop reachable? What's output of "ip addr show"
...
--
Jeremy Jackson
Coplanar Networks
(519)489-4903
http://www.coplanar.net
***@coplanar.net
Jared Gillis
17 years ago
Permalink
Oh wow, don't I feel silly now. That was the problem. After fixing that and removing the kernel default route the failover works as expected.

Is it possible to have the kernel default there as a backup-backup of sorts? I'm looking for functionality equivalent to:
If at least one BGP default exists, use it.
Else, use locally configured default.

Is this possible?
...
Dunc
17 years ago
Permalink
Post by Jared Gillis
Oh wow, don't I feel silly now. That was the problem. After fixing that
and removing the kernel default route the failover works as expected.
Is it possible to have the kernel default there as a backup-backup of
If at least one BGP default exists, use it.
Else, use locally configured default.
Is this possible?
Just configure a default with a distance greater than whatever routing
protocol you're using has.

ip route 0.0.0.0/0 x.x.x.x 250

ought to do it.

Cheers,

Dunc
...
Jeremy Jackson
17 years ago
Permalink
I was going to suggest using bgp neighbor 1.2.3.4 next-hop-self.

With Linux, you can have multiple routing tables (FIB). lartc.org
likely has the iproute2 HOWTOs or others on the subject. the key is the
"ip rule add" command.

You have create another routing table, with lower priority than the main
one, and put your static default route in there. If/when quagga puts a
default route in the main table, your static default will be "hidden"
since it's a lower priority.

# ip rule list
0: from all lookup local
10000: from 38.117.84.114 lookup rtrsafe <-- this is another trick, in
case quagga does something bad, this IP is always reachable from
internet.
32766: from all lookup main <-- quagga routes go here
32767: from all lookup default <-- exists by default, lower priority

# ip route show table default <-- special fallback if quagga dies
default via 70.47.139.253 dev nas0

# ip route show table main <-- these are made by zebra/ospfd (multipath)
default proto zebra metric 10
nexthop via 70.47.139.249 dev nas0 weight 1
nexthop via 70.47.139.253 dev nas0 weight 1
Post by Jared Gillis
Oh wow, don't I feel silly now. That was the problem. After fixing that and removing the kernel default route the failover works as expected.
If at least one BGP default exists, use it.
Else, use locally configured default.
Is this possible?
--
Jeremy Jackson
Coplanar Networks
(519)489-4903
http://www.coplanar.net
***@coplanar.net
Jared Gillis
17 years ago
Permalink
Excellent info, thanks very much!
I think this will probably help fix the "save-ass" default route issue.
...
Lennart Sorensen
17 years ago
Permalink
Post by Jared Gillis
Oh wow, don't I feel silly now. That was the problem. After fixing that and
removing the kernel default route the failover works as expected.
Is it possible to have the kernel default there as a backup-backup of
If at least one BGP default exists, use it.
Else, use locally configured default.
Is this possible?
As far as I understand linux routing, you can't have that.

I suppose if zebra was willing to insert routes that already had a
kernel route installed but at a different priority then a higher
priority route could override the kernel and if all options fail then
the kernel would onl have the original kernel route left and use that.
I haven't seen a way in quagga to make it do that though.
--
Len Sorensen
Jeremy Jackson
17 years ago
Permalink
Post by Lennart Sorensen
Post by Jared Gillis
Is it possible to have the kernel default there as a backup-backup of
If at least one BGP default exists, use it.
Else, use locally configured default.
Is this possible?
As far as I understand linux routing, you can't have that.
If you read some of the earlier replies you can see some examples of
people using multiple routing tables. Linux has a config option for
that, at least Ubuntu has it enabled in their distro.
--
Jeremy Jackson
Coplanar Networks
(519)489-4903
http://www.coplanar.net
***@coplanar.net
Lennart Sorensen
17 years ago
Permalink
Post by Jeremy Jackson
If you read some of the earlier replies you can see some examples of
people using multiple routing tables. Linux has a config option for
that, at least Ubuntu has it enabled in their distro.
Yeah. I haven't realized you could do it that way. Rather ugly
solution too in my opinion but I guess there isn't any way to make
multiple routing tables pretty.
--
Len Sorensen
Stegen Smith
17 years ago
Permalink
Hey Jared,

In what order did zebra and bgpd start up? I've noticed on some of
our systems that use zebra/bgpd that if the bgpd starts up before
zebra, then the fib and rib tables get a little wonky and zebra won't
know to insert the 0/0. Just a shot in the dark if the other options
don't work. ;)

stegen
...
Jeremy Jackson
17 years ago
Permalink
Post by Stegen Smith
Hey Jared,
In what order did zebra and bgpd start up? I've noticed on some of
our systems that use zebra/bgpd that if the bgpd starts up before
zebra, then the fib and rib tables get a little wonky and zebra won't
know to insert the 0/0. Just a shot in the dark if the other options
don't work. ;)
If your bgpd is starting before zebra, then your system's startup
scripts are broken. Debian/Ubuntu for example has a priority sequence
for the various quagga daemons when starting/stopping
with /etc/init.d/quagga script.
--
Jeremy Jackson
Coplanar Networks
(519)489-4903
http://www.coplanar.net
***@coplanar.net
Jared Gillis
17 years ago
Permalink
Strange that this was brought up. Now that I've got a solution to the backup default route issue, I was going to dig into this issue which I discovered.
My init scripts are fine and zebra/bgpd start up in the proper order, but I find that if I restart the bgpd process without touching zebra, zebra loses contact with bgpd and thus all the bgpd routes.
I was about to start digging around for some kind of way to make sure that both processes are restarted simultaneously, or that they can find each other again after a HUP, or at the very least, monitor the communication between them to make sure that I can manually intervene in the case where they de-sync.
...
--
Jared Gillis - ***@corp.sonic.net Sonic.net, Inc.
Network Operations 2260 Apollo Way
707.522.1000 (Voice) Santa Rosa, CA 95407
707.547.3400 (Support) http://www.sonic.net/
Loading...