25 years ago, I worked for a small ISP (back when there was such a thing.) When I started there, we had one upstream ISP. I was charged with getting us multihomed. I found some tutorials written by Avi Freedman (1). I don't know what I would have done without him. He made an intimidating topic approachable. Thanks to him, I got us a /20 from ARIN and advertised our routes to two different peers. It was fascinating to learn how it all worked. And the more I learned about it, the more amazed I was that it worked at all.
Like the human body the more you study the Internet the more amazing it is not that it sometimes breaks, but that it works at all. Especially for video/phone/etc.
Glad the content was helpful, I have links to some of them at avi.net (tutorials and old Boardwatch articles).
I swear my motive was pure (frustration with the content out there) but it was easy to see back then that helping people out with good content yields rewards ("Can I buy a T1?") or ("Come run my big global network"). So I still encourage everyone to write about what's confusing and frustrating...
chriscjcj 5 hours ago [-]
I think I sent you an e-mail back then, but let me take this opportunity to heap a bunch of praise and gratitude upon you in front of God and everybody. I bought several books on BGP at the time and slogged through them, but it was intimidating and I had a lot of trouble putting the pieces together. Your tutorials were simple, straightforward and helped me an awful lot. And the other thing about the way you wrote them that I really appreciated was that you didn't make me feel inferior for not knowing WTF I was doing. Thanks Avi. From time to time, I think back on the challenges I had in my life/career and I'm grateful for the people who helped me. You're definitely on that list, sir.
nickstinemates 16 hours ago [-]
I used to work for Avi at Kentik. He is a smart, nice person and remembered writing these articles fondly to help people out!
aspenmayer 6 hours ago [-]
> Avi
Somewhat off-topic, I’ve noticed on HN that sometimes comments end with single word on a newline, like you have done here.
Did your comment get truncated somehow, or did you mess up a copy/paste, or what?
Has anyone else noticed this? Is this a telltale sign of generated comments, or other tool usage?
I’ve only noticed this on HN, and maybe Reddit once? It happens somewhat more regularly than I can attribute to mere coincidence or accident.
kortilla 20 hours ago [-]
Nice article.
That flapping from EpicUp 140.99.244.0/23 prefix should have been subject to route dampening. This is per peer or per prefix rate limiting typically enforced on all peers by ISPs to prevent this exact issue of a single prefix making up a significant portion of the global BGP churn.
I’m unconvinced of the correlation between the updates that the author attributed to knock on effects. It would be pretty janky to have your advertisements be based on the path to other autonomous systems’ prefixes, especially unstable ones.
I don’t think there is a 40 minute periodicity either (at least there wasn’t 8 years ago when I was deep in the BGP world). Smells like what this dataset happened to show either by luck or because of the network the author was getting the BGP feed from.
If you dig into the data and look at which AS’s and prefixes are experiencing changes, you’ll find it’s all over the place and there isn’t really any bigger pattern.
On any given day there are usually a few noisy ISPs because of bad circuits or misconfigurations. Then there are new prefixes flapping in and out as a new thing is brought online for the first time, etc. Then sprinkle in path changes for regular draining maintenance, etc.
It’s simultaneously both fascinating and a little horrifying how a little ISP in Kansas experiencing a fiber consuming backhoe shows up on routers in Perth. Yet the frequency of updates is kept to <10hz globally through tons of hand tuned policies.
benjojo12 15 hours ago [-]
Route dampening has mostly fallen out of fashion with networks these days.
Most setups were horribly misconfigured and (most) routers are no longer extremely CPU starved as they once were, That doesn't mean that it does not still exist of course, when I did bgp battleships ( https://blog.benjojo.co.uk/post/bgp-battleships ) I found that 3356 (at the time) was doing route dampening, so play had to be paused for a while.
kortilla 13 hours ago [-]
That seems crazy to me. What guardrails are there against a single hacked router pumping 10000 path changes/sec?
gjf 10 hours ago [-]
The direct peering to the router is likely going to have a bad time, but route advertisement interval I mention in the article is going to coalesce all of those updates together. Downstream peers would only see the one update every 30 seconds (or so).
throw0101a 20 hours ago [-]
If anyone wants to learn about BGP (especially day-to-day stuff for peering scenarios), the Network Startup Resource Center out of U.Oregon has a good series of videos going through things:
Based on some quick sleuthing, I would assume that the 0xff reserved BGP attribute is likely a huawei quirk. Almost all of the 0xff's visible to bgp.tools (hi) follow the same format as the one in the post, and some of those networks with them seem to be running huawei kit.
clbrmbr 7 hours ago [-]
I once designed and configured a satellite-microwave hybrid network for a large US customer with field offices around Borneo. I’ll never forget making the leased line handoffs back in Jakarta. I had exactly zero experience doing this so I googled around and read that BGP is what you would use to connect between our OSPF/UBNT net and their IGRP/Cisco corporate WAN. When asked to configure BGP on their routers, the guys from Tata were like “what do you think you are AT&T or something?” We did kind of feel that way until one season of lightning strikes took out most of our AirFibers.
CJefferson 1 days ago [-]
I learnt quite a few things I didn’t know about BGP from this article, probably most interesting how chaotic it is!
I’d definitely be interested to read some follow-ups, diving into more details.
zokier 16 hours ago [-]
What is the easiest way for average joe to get hands on BGP data? If I wanted to try do similar analysis and don't happen to have a friend at ISP.
I have had a project on the back burner for about a year now to offer a BGP feed via a websocket to facilitate people playing around and doing research without allowing them to accidently spew crap into the DFZ. Shoot me an email if you are interested and I'll try to get it spun up this week.
icedchai 14 hours ago [-]
Generally, your upstream won't allow you to spew crap. Route filters are in place. I run a small AS for hobbyist purposes and all my upstreams are locked down.
nhggfu 17 hours ago [-]
/me thinks to himself : real nice font on that page
motohagiography 8 hours ago [-]
> What is apparent in both the path and IP space changes over time is that there is some sort of cyclic behaviour in the IPv4 updates.
does this imply the internet has something akin to tides ?
Rendered at 08:37:02 GMT+0000 (UTC) with Wasmer Edge.
(1) http://avi.freedman.net/
Avi
Like the human body the more you study the Internet the more amazing it is not that it sometimes breaks, but that it works at all. Especially for video/phone/etc.
Glad the content was helpful, I have links to some of them at avi.net (tutorials and old Boardwatch articles).
I swear my motive was pure (frustration with the content out there) but it was easy to see back then that helping people out with good content yields rewards ("Can I buy a T1?") or ("Come run my big global network"). So I still encourage everyone to write about what's confusing and frustrating...
Somewhat off-topic, I’ve noticed on HN that sometimes comments end with single word on a newline, like you have done here.
Did your comment get truncated somehow, or did you mess up a copy/paste, or what?
Has anyone else noticed this? Is this a telltale sign of generated comments, or other tool usage?
I’ve only noticed this on HN, and maybe Reddit once? It happens somewhat more regularly than I can attribute to mere coincidence or accident.
That flapping from EpicUp 140.99.244.0/23 prefix should have been subject to route dampening. This is per peer or per prefix rate limiting typically enforced on all peers by ISPs to prevent this exact issue of a single prefix making up a significant portion of the global BGP churn.
I’m unconvinced of the correlation between the updates that the author attributed to knock on effects. It would be pretty janky to have your advertisements be based on the path to other autonomous systems’ prefixes, especially unstable ones.
I don’t think there is a 40 minute periodicity either (at least there wasn’t 8 years ago when I was deep in the BGP world). Smells like what this dataset happened to show either by luck or because of the network the author was getting the BGP feed from.
If you dig into the data and look at which AS’s and prefixes are experiencing changes, you’ll find it’s all over the place and there isn’t really any bigger pattern.
On any given day there are usually a few noisy ISPs because of bad circuits or misconfigurations. Then there are new prefixes flapping in and out as a new thing is brought online for the first time, etc. Then sprinkle in path changes for regular draining maintenance, etc.
It’s simultaneously both fascinating and a little horrifying how a little ISP in Kansas experiencing a fiber consuming backhoe shows up on routers in Perth. Yet the frequency of updates is kept to <10hz globally through tons of hand tuned policies.
Most setups were horribly misconfigured and (most) routers are no longer extremely CPU starved as they once were, That doesn't mean that it does not still exist of course, when I did bgp battleships ( https://blog.benjojo.co.uk/post/bgp-battleships ) I found that 3356 (at the time) was doing route dampening, so play had to be paused for a while.
* https://learn.nsrc.org/bgp
I’d definitely be interested to read some follow-ups, diving into more details.
* https://www.ripe.net/analyse/internet-measurements/routing-i...
* https://lukasz.bromirski.net/post/bgp-w-labie-3/
does this imply the internet has something akin to tides ?