[rancid] persistent alerts - but nothing was changed ... ?
Alan McKinnon
alan.mckinnon at gmail.com
Wed Jul 17 07:59:27 UTC 2013
On 17/07/2013 09:24, Wilkinson, Alex wrote:
> 0n Wed, Jul 17, 2013 at 03:49:50PM +1000, Alan McKinnon wrote:
>
> >On 17/07/2013 03:46, Wilkinson, Alex wrote:
> >> 0n Wed, Jul 17, 2013 at 06:53:45AM +1000, heasley wrote:
> >>
> >> >Tue, Jul 16, 2013 at 11:53:23AM +0800, Wilkinson, Alex:
> >> >> 0n Tue, Jul 16, 2013 at 12:47:55PM +1000, Roy wrote:
> >> >>
> >> >> >I don't know the box but the diff seems to indicate that the location of
> >> >> >mgmt0 and mgmt1 interfaces in the config file is changing. In the first
> >> >> >diff the interfaces are before Ethernet1/1 and in the second diff, they
> >> >> >are after Ethernet 1/20
> >> >>
> >> >> You are exactly right. When comparing the diffs via OpenGrok its very clear that
> >> >> the line locations consistently change causing a diff + rancid alert. If i'm not
> >> >> changing these devices and the line locations are supposedly changing - what
> >> >> could cause this ? rancid ?
> >> >
> >> >it wouldnt be rancid, it's the device itself. report the bug to the TAC.
> >>
> >> Something I have noticed is that for all my Nexus devices (7K, 5K, 4K) none of
> >> them have the 'RANCID-CONTENT-TYPE' of 'cisco-nx' but rather plain old 'cisco'.
> >> Could this be the reason behind the my problem ? And even if not, why would
> >> Rancid not be using cisco-nx automagically ? Or do I have to set it manually ?
> >
> >
> >You have to set it manually in router.db
> >
> >Rancid has no auto-detection of device type
>
> Ah, great! I changed all NX devices to cisco-nx. This seems to fix the false alert
> noise but now introduces a new problem for our Nexus 5000s e.g
>
> Trying to get all of the configs.
> nexus5k1-1: missed cmd(s): show cores vdc-all,show processes log vdc-all,show running-config
> nexus5k1-1: End of run not found
> !
> nexus5k1-2: missed cmd(s): show cores vdc-all,show processes log vdc-all,show running-config
> nexus5k1-2: End of run not found
> !
> nexus5k2-2: missed cmd(s): show cores vdc-all,show processes log vdc-all,show running-config
> nexus5k2-2: End of run not found
> !
> nexus5k2-1: missed cmd(s): show cores vdc-all,show processes log vdc-all,show running-config
> nexus5k2-1: End of run not found
>
> All other Nexus devices work fine now (7Ks, 4Ks).
>
> Any ideas ?
The failing commands are the last three, so I'd suspect the command just
before those - show debug.
Stuff to check:
1. Does clogin properly work and properly enable the login?
2. What is in the log files (${RANCIDDIR}/var/logs/*) for those devices?
3. Does the rancid user have proper permissions to run those commands?
(keep in mind that if you use tacacs for auth, that Nexus are *very*
different from IOS wrt authorization).
I found that the default behaviour in the nxrancid code was to quit the
script entirely on permission denied errors, which caused me huge
issues. So now I apply this patch (beware, it's long):
(my notes about what it does):
* The nexus parser is incomplete and "Permission denied" errors cause
the parser to fail and quit. We want to continue regardless and mark the
command as having failed.
* Oddly enough, "show fex" and "show module fex" are not supported out
of the box.
* Oddly enough, the routine to sanitize SNMP community strings is
commented out.
* Changed the code to redact tacacs keys
* Expanded password redaction regex to exclude "mpls ldp neighbor"
> --- nxrancid.orig 2012-02-28 12:21:51.000000000 +0200
> +++ nxrancid 2013-05-15 11:14:14.000000000 +0200
> @@ -174,7 +174,8 @@
> return(1) if /Line has invalid autocommand /;
> return(1) if /(Invalid input detected|Type help or )/;
> return(-1) if (/\% Invalid command at /);
> - return(-1) if (/\% Permission denied/);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> return(-1) if (/command authorization failed/i);
>
> if (/^Cisco Nexus Operating System/) { $type = "NXOS";}
> @@ -241,7 +242,8 @@
> return(1) if /Line has invalid autocommand /;
> return(1) if /(Invalid input detected|Type help or )/;
> return(1) if (/\% Invalid command at /);
> - return(-1) if (/\% Permission denied/);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> return(-1) if (/command authorization failed/i);
>
> /^Built By / && ProcessHistory("COMMENTS","","", "!Build: $_");
> @@ -264,7 +266,8 @@
> next if (/^(\s*|\s*$cmd\s*)$/);
> return(1) if /(Invalid input detected|Type help or )/;
> return(1) if (/\% Invalid command at /);
> - return(-1) if (/\% Permission denied/);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> return(-1) if (/command authorization failed/i);
>
> /^-+$/ && next; # Skip lines of all dashes.
> @@ -286,7 +289,8 @@
> return(1) if /Line has invalid autocommand /;
> return(1) if /(Invalid input detected|Type help or )/;
> return(1) if (/\% Invalid command at /);
> - return(-1) if (/\% Permission denied/);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> return(-1) if (/command authorization failed/i);
>
> s/ +$//; # Drop trailing ' '
> @@ -307,7 +311,8 @@
> return(1) if /Line has invalid autocommand /;
> return(1) if /(Invalid input detected|Type help or )/;
> return(1) if (/\% Invalid command at /);
> - return(-1) if (/\% Permission denied/);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> return(-1) if (/command authorization failed/i);
>
> s/ +$//; # Drop trailing ' '
> @@ -328,7 +333,8 @@
> return(1) if /Line has invalid autocommand /;
> return(1) if /(Invalid input detected|Type help or )/;
> return(-1) if (/\% Invalid command at /);
> - return(-1) if (/\% Permission denied/);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> return(-1) if (/command authorization failed/i);
>
> # Cut out CurTemp - drop the 2nd to last field.
> @@ -358,7 +364,8 @@
> return(1) if /Line has invalid autocommand /;
> return(1) if /(Invalid input detected|Type help or )/;
> return(1) if (/\% Invalid command at /);
> - return(-1) if (/\% Permission denied/);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> return(-1) if (/command authorization failed/i);
>
> # Cut out Actual Output/Draw.
> @@ -404,7 +411,8 @@
> return(1) if /(Invalid input detected|Type help or )/;
> return(1) if /Ambiguous command/i;
> return(-1) if (/\% Invalid command at /);
> - return(-1) if (/\% Permission denied/);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> return(-1) if (/command authorization failed/i);
>
> s/ variable = / = /;
> @@ -434,7 +442,8 @@
> return(1) if / is either not present or not formatted/;
> return(-1) if /\%Error calling/;
> return(-1) if /(: device being squeezed|ATA_Status time out)/i; # busy
> - return(-1) if (/\% Permission denied/);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> return(-1) if (/command authorization failed/i);
> return(1) if /(Open device \S+ failed|Error opening \S+:)/;
>
> @@ -460,7 +469,8 @@
> last if (/^$prompt/);
> next if (/^\s*$cmd\s*$/);
> return(1) if (/\% Invalid command at /);
> - return(-1) if (/\% Permission denied/);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> return(-1) if (/command authorization failed/i);
>
> s/(.*) \*$/$1/; # Drop a trailing '*'
> @@ -485,7 +495,8 @@
> return(1) if /Line has invalid autocommand /;
> return(1) if /(Invalid input detected|Type help or )/;
> return(1) if (/\% Invalid command at /);
> - return(-1) if (/\% Permission denied/);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> return(-1) if (/command authorization failed/i);
>
> if (/^(NAME: "[^"]*",)\s+(DESCR: "[^"]+")/) {
> @@ -532,7 +543,8 @@
> return(1) if /Line has invalid autocommand /;
> return(1) if /(Invalid input detected|Type help or )/;
> return(1) if (/\% Invalid command at /);
> - return(-1) if (/\% Permission denied/);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> return(-1) if (/command authorization failed/i);
> next if (/^Configuration last modified by/);
> # the pager can not be disabled per-session on the PIX
> @@ -568,7 +575,8 @@
> # newer releases (~12.1(9)) place the vlan config in the normal
> # configuration (write term).
> return(1) if ($type =~ /^(3550|4500)$/);
> - return(-1) if (/\% Permission denied/);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> return(-1) if (/command authorization failed/i);
> # the pager can not be disabled per-session on the PIX
> if (/^(<-+ More -+>)/) {
> @@ -594,7 +602,8 @@
> return(1) if /Line has invalid autocommand /;
> return(1) if /(Invalid input detected|Type help or )/;
> return(-1) if (/\% Invalid command at /);
> - return(-1) if (/\% Permission denied/);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> return(-1) if (/command authorization failed/i);
>
> /^No matching debug flags set$/ && next;
> @@ -619,7 +628,8 @@
> return(1) if /Line has invalid autocommand /;
> return(1) if /(Invalid input detected|Type help or )/;
> return(1) if (/\% Invalid command at /);
> - return(-1) if (/\% Permission denied/);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> return(-1) if (/command authorization failed/i);
>
> ProcessHistory("COMMENTS","","","!CORES: $_");
> @@ -639,7 +649,8 @@
> return(1) if /Line has invalid autocommand /;
> return(1) if /(Invalid input detected|Type help or )/;
> return(-1) if (/\% Invalid command at /);
> - return(-1) if (/\% Permission denied/);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> return(-1) if (/command authorization failed/i);
>
> ProcessHistory("COMMENTS","","","!PROC_LOGS: $_");
> @@ -648,6 +659,27 @@
> return(0);
> }
>
> +# This routine parses "show fex" and "show module fex"
> +sub ShowFex {
> + print STDERR " In ShowFex: $_" if ($debug);
> +
> + while (<INPUT>) {
> + tr/\015//d;
> + last if (/^$prompt/);
> + next if (/^(\s*|\s*$cmd\s*)$/);
> + return(1) if /Line has invalid autocommand /;
> + return(1) if /(Invalid input detected|Type help or )/;
> + return(1) if (/\% Invalid command at /);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> + return(-1) if (/command authorization failed/i);
> +
> + ProcessHistory("COMMENTS","","","!FEX: $_");
> + }
> + ProcessHistory("COMMENTS","","","!\n");
> + return(0);
> +}
> +
> # This routine processes a "write term"
> sub WriteTerm {
> print STDERR " In WriteTerm: $_" if ($debug);
> @@ -660,7 +692,8 @@
> return(1) if (/(Invalid input detected|Type help or )/i);
> return(-1) if (/\% Invalid command at /);
> return(0) if ($found_end); # Only do this routine once
> - return(-1) if (/\% Permission denied/);
> +# return(-1) if (/\% Permission denied/);
> + return(1) if (/\% Permission denied/);
> return(-1) if (/command authorization failed/i);
>
> # /Non-Volatile memory is in use/ && return(-1); # NvRAM is locked
> @@ -784,6 +822,10 @@
> # ProcessHistory("","","","! neighbor $1 password <removed>\n");
> # next;
> # }
> + if (/^\s*(.*?neighbor \S*) password / && $filter_pwds >= 1) {
> + ProcessHistory("","","","! $1 password <removed>\n");
> + next;
> + }
> # if (/^(ppp .* password) 7 .*/ && $filter_pwds >= 1) {
> # ProcessHistory("","","","!$1 <removed>\n"); next;
> # }
> @@ -929,18 +962,25 @@
> # }
> # next;
> # }
> -# if (/^(snmp-server community) (\S+)/) {
> -# if ($filter_commstr) {
> -# ProcessHistory("SNMPSERVERCOMM","keysort","$_","!$1 <removed>$'") && next;
> -# } else {
> -# ProcessHistory("SNMPSERVERCOMM","keysort","$_","$_") && next;
> -# }
> -# }
> + # Why was this commented out? It shows up in the raw text...
> + if (/^(snmp-server community) (\S+)/) {
> + if ($filter_commstr) {
> + ProcessHistory("SNMPSERVERCOMM","keysort","$_","!$1 <removed>$'") && next;
> + } else {
> + ProcessHistory("SNMPSERVERCOMM","keysort","$_","$_") && next;
> + }
> + }
> # # prune tacacs/radius server keys
> # if (/^((tacacs|radius)-server\s(\w*[-\s(\s\S+])*\s?key) (\d )?\w+/
> # && $filter_pwds >= 1) {
> # ProcessHistory("","","","!$1 <removed>$'"); next;
> # }
> + # prune tacacs/radius server keys:
> + # tacacs-server host 196.23.0.13 key 7 "xxxxxxx" port 50 timeout 10
> + if (/^((tacacs|radius)-server.*?\bkey\b.*?) ".*?"(.*)/
> + && $filter_pwds >= 1) {
> + ProcessHistory("","","","!$1 <removed>$3\n"); next;
> + }
> # # order clns host statements
> # /^clns host \S+ (\S+)/ &&
> # ProcessHistory("CLNS","keysort","$1","$_") && next;
> @@ -1035,7 +1075,7 @@
> {'show boot' => 'ShowBoot'},
> {'dir bootflash:' => 'DirSlotN'},
> {'dir debug:' => 'DirSlotN'},
> - {'dir logflash:' => 'DirSlotN'},
> +# {'dir logflash:' => 'DirSlotN'},
> {'dir slot0:' => 'DirSlotN'},
> {'dir usb1:' => 'DirSlotN'},
> {'dir usb2:' => 'DirSlotN'},
> @@ -1048,6 +1088,8 @@
> {'show debug' => 'ShowDebug'},
> {'show cores vdc-all' => 'ShowCores'},
> {'show processes log vdc-all' => 'ShowProcLog'},
> + {'show module fex' => 'ShowFex'},
> + {'show fex' => 'ShowFex'},
> {'show running-config' => 'WriteTerm'},
> );
> # Use an array to preserve the order of the commands and a hash for mapping
>
> [edit] bin/rancid
>
> Removed dynamic address data from a description line for Ethernet interfaces.
> Expanded password redaction regex to exclude "mpls ldp neighbor"
>
> --- rancid.orig 2012-12-20 22:46:04.000000000 +0200
> +++ rancid 2012-12-20 22:48:51.000000000 +0200
> @@ -835,6 +835,10 @@
> /^AM79970 / && ProcessHistory("INT","","","!Interface: $_") && next;
> /^buffer size \d+ (Universal Serial: .*)/ &&
> ProcessHistory("INT","","","!\t$1\n") && next;
> + # Remove these dynamic addresses:
> + # !Interface: FastEthernet0/0, GT96K FE ADDR: 62AFB684, FASTSEND: 61579E4C, MCI_INDEX: 0
> + /^Hardware is (.*?)($| ADDR: .*| at 0x.*)/ &&
> + ProcessHistory("INT","","","!Interface: $INT$1\n") && next;
> /^Hardware is (.*)/ &&
> ProcessHistory("INT","","","!Interface: $INT$1\n") && next;
> /^(QUICC Serial unit \d),/ &&
> @@ -1741,8 +1745,8 @@
> ProcessHistory("LINE-PASS","","","!$1secret <removed>\n");
> next;
> }
> - if (/^\s*neighbor (\S*) password / && $filter_pwds >= 1) {
> - ProcessHistory("","","","! neighbor $1 password <removed>\n");
> + if (/^\s*(.*?neighbor.*?) password / && $filter_pwds >= 1) {
> + ProcessHistory("","","","! $1 password <removed>\n");
> next;
> }
> if (/^(\s*ppp .* hostname) .*/ && $filter_pwds >= 1) {
--
Alan McKinnon
alan.mckinnon at gmail.com
More information about the Rancid-discuss
mailing list