I've been having a really bizarre problem off and on the last week or two. Occasionally some of my ARCs will suddenly stop doing OSPF right. The routing table loses all OSPF routes, and the ARC stops announcing its routes via OSPF to our Ciscos. Disabling/enabling OSPF doesn't help, but most of the time (not all the time) rebooting the ARC will. One thing that seemed to trigger it was rebooting some of our Ciscos... and since that happens so infrequently (not rebooted since going to ARC 4.2 in fact) it's hard to track down. I don't know which one exactly, maybe our AS border router dying is what threw it off. It also happened last night when I installed a Cisco Catalyst 2924XL (replacing an old 10BaseT hub, finally!) which didn't involve rebooting the Ciscos, but involved having them unreachable for a minute or two as cables were switched and Spanning Tree did its thing... I have another ARC with no modems on the same LAN (for testing) and it initially did the same thing, but now I'm having trouble reproducing it. When it was screwed up one night, it fixed itself overnight, but it obviously took hours to do, which kinda defeats the purpose of OSPF. I know this is really vague, but problem is, I don't really know where to start looking for this, because I didn't document very well what my current OSPF settings are. (A bulk-config-to-ASCII utility would be REALLY HELPFUL HERE :) I could reverse-engineer HARM and try to get a dump of that part of the config via SNMP somehow, I guess, but... yuck. Anyway, I'm also not a real in-depth OSPF guru yet. Has anyone seen anything like this before or know of any particular settings that I might want to check? This is driving me nuts (not to mention my customers) and I'm not sure where to start. A quick look at some of the 'show' screens -- this is with the card actually routing correctly:
show ospf global
GLOBAL OSPF SETTINGS Router ID: 206.240.130.12 Administrative Status: ENABLED OSPF Version Number: 2 Area Border Router Status: DISABLED Autonomous System Border Router Status: ENABLED Type-of-service (TOS) Support: OFF External LSDB Limit: -1 Multicast Extensions Support: OFF Exit Overflow Interval: 0 Router's Support for OSPF Demand Routing: OFF OSPF Traps: DISABLED
show ospf area 0.0.0.0
OSPF AREA SETTINGS Area ID: 0.0.0.0 OSPF Area Type: TRANSIT Area Status: ENABLED
list ospf recei
OSPF Receive Policies: Address/Mask Action Routing Preference 199.077.100.000/23 Accept Pref1 206.240.130.000/23 Accept Pref1 208.006.168.000/22 Accept Pref1
list ospf send
OSPF Send Policies: Source Address/Mask Action LOCAL 199.077.100.000/23 Advertise LOCAL 206.240.130.000/23 Advertise LOCAL 208.006.168.000/22 Advertise
list ospf int OSPF INTERFACE TABLE IfIpAddr/IfInd AreaId IfType AdminStat IfState Metric 206.240.130.12 0.0.0.0 BC ENABLED OtherDsgRtr 10
Mike Andrews (MA12) * mandrews@dcr.net * http://www.bit0.com/ VP, sysadmin, & network guy, Digital Crescent Inc, Frankfort KY Internet services for Frankfort, Lawrenceburg, Owenton, & Shelbyville "With sufficient thrust, pigs fly just fine." -- RFC 1925 - To unsubscribe to usr-tc, send an email to "majordomo@xmission.com" with "unsubscribe usr-tc" in the body of the message. For information on digests or retrieving files and old messages send "help" to the same address. Do not use quotes in your message.
Thus spake Mike Andrews
I've been having a really bizarre problem off and on the last week or two. Occasionally some of my ARCs will suddenly stop doing OSPF right. The routing table loses all OSPF routes, and the ARC stops announcing its routes via OSPF to our Ciscos. Disabling/enabling OSPF doesn't help, but most of the time (not all the time) rebooting the ARC will.
One thing that seemed to trigger it was rebooting some of our Ciscos... and since that happens so infrequently (not rebooted since going to ARC 4.2 in fact) it's hard to track down. I don't know which one exactly, maybe our AS border router dying is what threw it off.
Hrmm...something to check...not sure if this will have any signficance...is the settings/values for the designated router on your ethernet(s) when this happens. I suspect your Cisco's were the designated routers initially, and when they went away, one of your Arc's got promoted to that...who knows what happened when your cisco's came back. When I was first playing with 4.2.29, I noticed a little bit of odd behavior with the Arc's wrt designated routers...but then when I set up a test lab environment for it, I couldn't reproduce it. What I saw was then I brought the Arc up on 4.2.29, it would take over the designated router functionality somehow...which would be against the OSPF spec, but possible for it to happen. I haven't played with OSPF on the arc's since then in any great depth...I've got one up and running on 4.2.32 at this point, but haven't really done anything with it. Maybe this will give you an idea of something to look at. Switching designated routers is definitely something you want to avoid if at all possible because its going to cause a resyncronization of LSA databases and a recalc of most (all?) of the shortest path first calculations...so...this *can* cause up to a couple of minutes of your routing being hosed as potentially every router on your network recalcs its OSPF routes. -- Jeff McAdams Email: jeffm@iglou.com Head Network Administrator Voice: (502) 966-3848 IgLou Internet Services (800) 436-4456 - To unsubscribe to usr-tc, send an email to "majordomo@xmission.com" with "unsubscribe usr-tc" in the body of the message. For information on digests or retrieving files and old messages send "help" to the same address. Do not use quotes in your message.
This exact problem has happened to me several times and it was NOT caused by the ARC deciding to become the DR. I have a small network of about 13 Lucent portmasters, 3 USR TC chassis, one Cisco 4500 and one Cisco 4700 router. ALL of the termservers are set with OSPF priority 0, meaning they will never become a DR or BDR. The Cisco 4500 and 4700 OSPF priorities were left as-is leaving one to become the DR and the other to become the BDR. If I rebooted or otherwise disturbed the DR the USR TC chassis would lose all their OSPF routing. The Lucent equipment did not have a problem at all. Getting the OSPF back on the USRs required a reboot of every chassis. For now I have set one of the Ciscos to priority 0 so that it will never become a BDR or DR. It seems as though the HARC cards have a problem recognizing a change of a BDR to a DR while the rest of my equipment does not. If this problem is indeed being caused by an ARC deciding to become a DR then it would seem to be a bug. In my case this should not be happening as all the HARCs are set to NOT participate in DR/BDR elections. Mike McHenry Systems Administrator MinnNet Communications, Inc. -----Original Message----- From: owner-usr-tc@lists.xmission.com [mailto:owner-usr-tc@lists.xmission.com]On Behalf Of Jeff Mcadams Sent: Monday, October 11, 1999 8:35 PM To: usr-tc@lists.xmission.com Subject: Re: (usr-tc) arcs and ospf Thus spake Mike Andrews
I've been having a really bizarre problem off and on the last week or two. Occasionally some of my ARCs will suddenly stop doing OSPF right. The routing table loses all OSPF routes, and the ARC stops announcing its routes via OSPF to our Ciscos. Disabling/enabling OSPF doesn't help, but most of the time (not all the time) rebooting the ARC will.
One thing that seemed to trigger it was rebooting some of our Ciscos... and since that happens so infrequently (not rebooted since going to ARC 4.2 in fact) it's hard to track down. I don't know which one exactly, maybe our AS border router dying is what threw it off.
Hrmm...something to check...not sure if this will have any signficance...is the settings/values for the designated router on your ethernet(s) when this happens. I suspect your Cisco's were the designated routers initially, and when they went away, one of your Arc's got promoted to that...who knows what happened when your cisco's came back. When I was first playing with 4.2.29, I noticed a little bit of odd behavior with the Arc's wrt designated routers...but then when I set up a test lab environment for it, I couldn't reproduce it. What I saw was then I brought the Arc up on 4.2.29, it would take over the designated router functionality somehow...which would be against the OSPF spec, but possible for it to happen. I haven't played with OSPF on the arc's since then in any great depth...I've got one up and running on 4.2.32 at this point, but haven't really done anything with it. Maybe this will give you an idea of something to look at. Switching designated routers is definitely something you want to avoid if at all possible because its going to cause a resyncronization of LSA databases and a recalc of most (all?) of the shortest path first calculations...so...this *can* cause up to a couple of minutes of your routing being hosed as potentially every router on your network recalcs its OSPF routes. -- Jeff McAdams Email: jeffm@iglou.com Head Network Administrator Voice: (502) 966-3848 IgLou Internet Services (800) 436-4456 - To unsubscribe to usr-tc, send an email to "majordomo@xmission.com" with "unsubscribe usr-tc" in the body of the message. For information on digests or retrieving files and old messages send "help" to the same address. Do not use quotes in your message. - To unsubscribe to usr-tc, send an email to "majordomo@xmission.com" with "unsubscribe usr-tc" in the body of the message. For information on digests or retrieving files and old messages send "help" to the same address. Do not use quotes in your message.
Thus spake Mike McHenry
This exact problem has happened to me several times and it was NOT caused by the ARC deciding to become the DR. I have a small network of about 13 Lucent portmasters, 3 USR TC chassis, one Cisco 4500 and one Cisco 4700 router. ALL of the termservers are set with OSPF priority 0, meaning they will never become a DR or BDR. The Cisco 4500 and 4700 OSPF priorities were left as-is leaving one to become the DR and the other to become the BDR.
If I rebooted or otherwise disturbed the DR the USR TC chassis would lose all their OSPF routing. The Lucent equipment did not have a problem at all. Getting the OSPF back on the USRs required a reboot of every chassis. For now I have set one of the Ciscos to priority 0 so that it will never become a BDR or DR. It seems as though the HARC cards have a problem recognizing a change of a BDR to a DR while the rest of my equipment does not.
If this problem is indeed being caused by an ARC deciding to become a DR then it would seem to be a bug. In my case this should not be happening as all the HARCs are set to NOT participate in DR/BDR elections.
Sounds like you've played with it considerably more than myself at least, I'll defer to your experience with this code. I had just played with it enough to know that it puked mightily when something happened to the DR...I hadn't played enough to know specifically what caused it. Sounds like you've dug into it far enough that you know more specifics. -- Jeff McAdams Email: jeffm@iglou.com Head Network Administrator Voice: (502) 966-3848 IgLou Internet Services (800) 436-4456 - To unsubscribe to usr-tc, send an email to "majordomo@xmission.com" with "unsubscribe usr-tc" in the body of the message. For information on digests or retrieving files and old messages send "help" to the same address. Do not use quotes in your message.
Mike Andrews writes...
I've been having a really bizarre problem off and on the last week or two. Occasionally some of my ARCs will suddenly stop doing OSPF right. The routing table loses all OSPF routes, and the ARC stops announcing its routes via OSPF to our Ciscos. Disabling/enabling OSPF doesn't help, but most of the time (not all the time) rebooting the ARC will.
One thing that seemed to trigger it was rebooting some of our Ciscos... and since that happens so infrequently (not rebooted since going to ARC 4.2 in fact) it's hard to track down. I don't know which one exactly, maybe our AS border router dying is what threw it off.
. . .
Has anyone seen anything like this before or know of any particular settings that I might want to check? This is driving me nuts (not to mention my customers) and I'm not sure where to start.
Make sure your tc's aren't becoming the DR. To check, do a "list ospf nei", see if any of them are in any state other than "two way". To fix, do a "set ospf interface THE_OSPF_INTERFACE router_priority 0" on each of them and reboot. -- Aaron Nabil - To unsubscribe to usr-tc, send an email to "majordomo@xmission.com" with "unsubscribe usr-tc" in the body of the message. For information on digests or retrieving files and old messages send "help" to the same address. Do not use quotes in your message.
|-----Original Message----- |From: owner-usr-tc@lists.xmission.com |[mailto:owner-usr-tc@lists.xmission.com]On Behalf Of Aaron Nabil |Sent: Tuesday, October 12, 1999 7:22 AM |To: usr-tc@lists.xmission.com |Subject: Re: (usr-tc) arcs and ospf | | |Mike Andrews writes... |>I've been having a really bizarre problem off and on the last week or two. |>Occasionally some of my ARCs will suddenly stop doing OSPF right. The |>routing table loses all OSPF routes, and the ARC stops announcing its |>routes via OSPF to our Ciscos. Disabling/enabling OSPF doesn't help, but |>most of the time (not all the time) rebooting the ARC will. |> |>One thing that seemed to trigger it was rebooting some of our Ciscos... |>and since that happens so infrequently (not rebooted since going to ARC |>4.2 in fact) it's hard to track down. I don't know which one exactly, |>maybe our AS border router dying is what threw it off. |> |> . . . |> |>Has anyone seen anything like this before or know of any particular |>settings that I might want to check? This is driving me nuts (not to |>mention my customers) and I'm not sure where to start. | |Make sure your tc's aren't becoming the DR. To check, do a "list ospf nei", |see if any of them are in any state other than "two way". To fix, do a |"set ospf interface THE_OSPF_INTERFACE router_priority 0" on each of |them and reboot. | That recomendation is actually in the release notes for 4.2.32... -M - To unsubscribe to usr-tc, send an email to "majordomo@xmission.com" with "unsubscribe usr-tc" in the body of the message. For information on digests or retrieving files and old messages send "help" to the same address. Do not use quotes in your message.
On Tue, 12 Oct 1999, Mike Wronski wrote:
|Mike Andrews writes... |>I've been having a really bizarre problem off and on the last week or two. |>Occasionally some of my ARCs will suddenly stop doing OSPF right. The |>routing table loses all OSPF routes, and the ARC stops announcing its |>routes via OSPF to our Ciscos. Disabling/enabling OSPF doesn't help, but |>most of the time (not all the time) rebooting the ARC will. |> |>One thing that seemed to trigger it was rebooting some of our Ciscos... |>and since that happens so infrequently (not rebooted since going to ARC |>4.2 in fact) it's hard to track down. I don't know which one exactly, |>maybe our AS border router dying is what threw it off. |> |> . . . |> |>Has anyone seen anything like this before or know of any particular |>settings that I might want to check? This is driving me nuts (not to |>mention my customers) and I'm not sure where to start. | |Make sure your tc's aren't becoming the DR. To check, do a "list ospf nei", |see if any of them are in any state other than "two way". To fix, do a |"set ospf interface THE_OSPF_INTERFACE router_priority 0" on each of |them and reboot. |
That recomendation is actually in the release notes for 4.2.32...
OK, that's three votes for DR/BDR designation being the problem. :) And dumb me for not paying closer attention to the release notes. Basically I had left every single router in my network set to the default of priority 0. I've just set my border Cisco to 50, the next fastest Cisco to 40, all the 2500's to 20, and left the TC's and two FreeBSD machines running gated set to 0. I rebooted what was the DR at the time (I know, coulda just disabled/reenabled OSPF, but the customers on it didn't notice) and didn't have any problems with the TC's. We'll see... Mike Andrews (MA12) * mandrews@dcr.net * http://www.bit0.com/ VP, sysadmin, & network guy, Digital Crescent Inc, Frankfort KY Internet services for Frankfort, Lawrenceburg, Owenton, & Shelbyville "With sufficient thrust, pigs fly just fine." -- RFC 1925 - To unsubscribe to usr-tc, send an email to "majordomo@xmission.com" with "unsubscribe usr-tc" in the body of the message. For information on digests or retrieving files and old messages send "help" to the same address. Do not use quotes in your message.
Can anyone tell me why a Chassis all of a sudden would simply shut itself down? There are no HIGH Temps or power fluctuations, it just shuts down... It is configured with 5 DSPs/HiperArc/NMC/ and 1 70amp PS It is plugged into a BCPro 850. Could this be a sign of a power supply getting ready to "crap the bed"? Thanks in advance! ============================================================================== Phillip Ferraro WorldNet Access, Inc pferraro@wna-linknet.com Onslow County's PREMIER InterNet Service Voice (910) 346-0835 824 Gumbranch Square, Suite Q FAX (910) 455-1933 Jacksonville, Nc 28540-6269 ============================================================================== - To unsubscribe to usr-tc, send an email to "majordomo@xmission.com" with "unsubscribe usr-tc" in the body of the message. For information on digests or retrieving files and old messages send "help" to the same address. Do not use quotes in your message.
Thus spake Mike Wronski
|Mike Andrews writes... |>I've been having a really bizarre problem off and on the last week or two. |>Occasionally some of my ARCs will suddenly stop doing OSPF right. The |>routing table loses all OSPF routes, and the ARC stops announcing its |>routes via OSPF to our Ciscos. Disabling/enabling OSPF doesn't help, but |>most of the time (not all the time) rebooting the ARC will.
|>One thing that seemed to trigger it was rebooting some of our Ciscos... |>and since that happens so infrequently (not rebooted since going to ARC |>4.2 in fact) it's hard to track down. I don't know which one exactly, |>maybe our AS border router dying is what threw it off.
|>Has anyone seen anything like this before or know of any particular |>settings that I might want to check? This is driving me nuts (not to |>mention my customers) and I'm not sure where to start.
|Make sure your tc's aren't becoming the DR. To check, do a "list ospf nei", |see if any of them are in any state other than "two way". To fix, do a |"set ospf interface THE_OSPF_INTERFACE router_priority 0" on each of |them and reboot.
That recomendation is actually in the release notes for 4.2.32...
There are problems with the DR/BDR code in 4.2.x then? Hrmm...could suck if you don't have 2 other OSPF speakers on a subnet...if they reboot, then the Arc would become DR, and when they come back up, they won't take over from it. :/ Would require a reboot of possibly two Arcs for get the DR to be something other than an Arc in that situation. :/ -- Jeff McAdams Email: jeffm@iglou.com Head Network Administrator Voice: (502) 966-3848 IgLou Internet Services (800) 436-4456 - To unsubscribe to usr-tc, send an email to "majordomo@xmission.com" with "unsubscribe usr-tc" in the body of the message. For information on digests or retrieving files and old messages send "help" to the same address. Do not use quotes in your message.
participants (6)
-
Aaron Nabil -
Jeff Mcadams -
Mike Andrews -
Mike McHenry -
Mike Wronski -
pferraro＠wna-linknet.com