[rancid] F5 Rancid | BIGIP 12.0 | missed cmd(s)
Daniel Kerse
daniel.kerse at gmail.com
Mon Sep 12 12:40:10 UTC 2016
Hello Shaun et.al.,
Are you playing with the TERM environment variable at all?
Have you tried enabling debug on your cron job and then (re)checking:
1. the logs?
2. the resulting .raw files?
I'm also running version 3.2 (the latest available under EPEL) with
some F5s of various vintage.
Firstly to recap what is already known by this mailing list:
- circa BIG-IP v9.x -> v10.x the tmsh shell was introduced in addition
to the pre-existing bigpipe.
- circa BIG-IP v10.x -> v11.x the old bigpipe shell was removed.
- (At least for BIG-IP v11.x and earlier) you must set TERM=vt100-w in
order to run some commands. This problem would only become apparent
when running via CRON, When running rancid manually at the CLI
everything looks honky dory.. In my case the following commands would
fail (via CRON):
"ls --full-time --color=never /config/ssl/ssl.crt"
"ls --full-time --color=never /config/ssl/ssl.key"
"tmsh show /net route static"
- Someone mentioned that you need to run "tmsh -q -c 'cd / \;list
recursive" to get everything out of the partitions.
- Was this circa v11.x? I forget. I need to do more research here
as I've almost got around that issue that had people setting up shell
scripts on their F5s to make it work... more on that later.
Now here's the interesting thing I've noticed on v12.x . Perhaps this
is widely known in the F5 community but I haven't read about it
anywhere.
- Somewhere around BIG-IP v11.x -> v12.x the vt100-w terminal started
misbehaving.
- If I have TERM set to vt100-w and enable debug I see this error
message in the logs:
"Warning, can't fully initialize terminal, TERM is set to
"vt100-w", status (0)"
- Now I think again if you run via CLI it might appear to work OK
but not via CRON.
So now you end up in a difficult situation in BIG-IP v12:
- TERM=vt100-w fails via CRON because it's broken
- TERM=network (or similar) fails via CRON because some commands
require it to stop the junk showing up mid-line.
- Either case works via CLI.
So you can end up with a rather fiddly rancid.types.conf file.
Something like this (work in progress):
# F5s running 9.3.1 seem to lack tmsh support so we will use a
(slightly modified) version of f5rancid
bigip-v9;script;f5rancid-v9
bigip-v9;login;clogin
# F5s running 11.4.1 and 12.0.0 seem to lack bigpipe support.
# For some reason turning off debug flag causes problems for "tmsh -q
-c 'cd / \;list recursive'" I don't know why.
bigip-v10-11;script;rancid -t bigip-v10-11
bigip-v10-11;login;clogin
bigip-v10-11;module;bigip
bigip-v10-11;inloop;bigip::inloop
bigip-v10-11;command;bigip::ShowVersion;tmsh show /sys version
bigip-v10-11;command;bigip::ShowHardware;tmsh show /sys hardware
bigip-v10-11;command;bigip::ShowLicense;tmsh show /sys license
bigip-v10-11;command;bigip::ShowSslCrt;ls --full-time --color=never
/config/ssl/ssl.crt
bigip-v10-11;command;bigip::ShowSslKey;ls --full-time --color=never
/config/ssl/ssl.key
bigip-v10-11;command;bigip::ShowZebOSconf;cat /config/ZebOS.conf
bigip-v10-11;command;bigip::ShowZebOSsockets;lsof -i :179
bigip-v10-11;command;bigip::ShowRouteStatic;tmsh show /net route static
# This one seems to get confused
#bigip-v10-11;command;bigip::WriteTerm;tmsh -q -c 'cd / \;list recursive'
bigip-v10-11;command;bigip::WriteTerm;tmsh -q list
# F5s running 12.0.0 seem to lack support for vt100-w terminal type.
# Error Message:
# "Warning, can't fully initialize terminal, TERM is set to "vt100-w",
status (0)"
# As a result ShowSslCrt, ShowSslKey and ShowRouteStatic don't work.
# For some reason turning off debug flag causes problems for "tmsh -q
-c 'cd / \;list recursive'" I don't know why.
bigip-v12;script;rancid -dt bigip-v12
bigip-v12;login;clogin
bigip-v12;module;bigip12
bigip-v12;inloop;bigip12::inloop
bigip-v12;command;bigip12::ShowVersion;tmsh show /sys version
bigip-v12;command;bigip12::ShowHardware;tmsh show /sys hardware
bigip-v12;command;bigip12::ShowLicense;tmsh show /sys license
#bigip-v12;command;bigip12::ShowSslCrt;ls --full-time --color=never
/config/ssl/ssl.crt
#bigip-v12;command;bigip12::ShowSslKey;ls --full-time --color=never
/config/ssl/ssl.key
bigip-v12;command;bigip12::ShowZebOSconf;cat /config/ZebOS.conf
bigip-v12;command;bigip12::ShowZebOSsockets;lsof -i :179
#bigip-v12;command;bigip12::ShowRouteStatic;tmsh show /net route static
#bigip-v12;command;bigip12::WriteTerm;tmsh -q -c 'cd / \;list recursive'
bigip-v12;command;bigip12::WriteTerm;tmsh -q list
# Huawei support not provided out of the box, added after the fact.
# Let me know if you want me to post this, it's a bit OT for this thread..
huawei;script;rancid -t huawei
huawei;login;hulogin
huawei;module;huawei
huawei;inloop;huawei::inloop
huawei;command;huawei::DisplayVersion;display version
huawei;command;huawei::DisplayPatchInfo;display patch-information
huawei;command;huawei::DisplayDevice;display esn
huawei;command;huawei::DisplayDevice;display device
huawei;command;huawei::DisplayDevice;display device manufacture-info
huawei;command;huawei::DisplayDevice;display device pic-status
huawei;command;huawei::DisplayElabel;display device elabel
huawei;command;huawei::DisplayElabel;display elabel
huawei;command;huawei::DisplayTransceiver;display interface transceiver
huawei;command;huawei::DisplayLicense;display license
huawei;command;huawei::WriteTerm;display current-configuration
With also the following customizations:
1) To make "tmsh -q -c 'cd / \;list recursive'" do something useful
the following change to rancid.pm is required, otherwise the split is
too aggressive.
*******
< my($type, $directive, $value, $value2) = split('\;');
---
> #my($type, $directive, $value, $value2) = split('\;');
> my($type, $directive, $value, $value2) = split('\;', $_, 4);
*******
2) Following changes made to bigip.pm (from rancid 3.5). There might
be a bug here still as uncommenting the ";tmsh -q -c 'cd / \;list
recursive'" line above seems to lead to unnecessary reattempts at
config collection - but only when debug is disabled for some reason.
*******
5c5
< ## rancid 3.5
---
> ## rancid 3.5 plus some customisations.
66c66,67
< use rancid 3.5;
---
> #use rancid 3.5;
> use rancid 3.2;
74c75
< $ENV{'TERM'} = "vt100";
---
> $ENV{'TERM'} = "vt100-w";
183a185
> s/^\ \ ([0-9]+)(\ +).*Air\ Outlet/ $1$2REMOVED Air Outlet/i;
184a187,188
> s/^\ \ ([0-9]+)(\ +).*ADM1026/ $1$2REMOVED ADM1026/i;
> s/^\ \ ([0-9]+)(\ +).*Main\ board(.*)$/ $1$2REMOVED Main board$3/i;
185a190
> s/^\ \ ([0-9]+)(\ +)[0-9]+\ +[0-9]+\ +[0-9]+/ $1$2REMOVED REMOVED REMOVED/;
218a224,272
> # This routine parses "ls --full-time --color=never /config/ssl/ssl.crt"
> sub ShowSslCrt {
> my($INPUT, $OUTPUT, $cmd) = @_;
> my($line) = (0);
> print STDERR " In ShowSslCrt: $_" if ($debug);
>
> while (<$INPUT>) {
> tr/\015//d;
> # v9 software license does not have CR at EOF
> s/^#-+($prompt.*)/$1/;
> last if (/^$prompt/);
> next if (/^(\s*|\s*$cmd\s*)$/);
> return(1) if /^\s*\^\s*$/;
> return(1) if /(Invalid input detected|Type help or )/;
> return(-1) if (/command authorization failed/i);
>
> if (!$line++) {
> ProcessHistory("ShowSslCrt","","","#\n#/config/ssl/ssl.crt:\n");
> }
> ProcessHistory("ShowSslCrt","","","# $_") && next;
> }
> return(0);
> }
>
> # This routine parses "ls --full-time --color=never /config/ssl/ssl.key"
> sub ShowSslKey {
> my($INPUT, $OUTPUT, $cmd) = @_;
> my($line) = (0);
> print STDERR " In ShowSslKey: $_" if ($debug);
>
> while (<$INPUT>) {
> tr/\015//d;
> # v9 software license does not have CR at EOF
> s/^#-+($prompt.*)/$1/;
> last if (/^$prompt/);
> next if (/^(\s*|\s*$cmd\s*)$/);
> return(1) if /^\s*\^\s*$/;
> return(1) if /(Invalid input detected|Type help or )/;
> return(-1) if (/command authorization failed/i);
>
> if (!$line++) {
> ProcessHistory("ShowSslKey","","","#\n#/config/ssl/ssl.key:\n");
> }
> ProcessHistory("ShowSslKey","","","# $_") && next;
> }
> return(0);
> }
>
>
290a345,347
> return (1) if (/Syntax Error: unexpected argument/);
> return (0) if ($found_end); # Only run this routine once.
>
297c354
< $found_end++;
---
> $found_end = 1;
*******
Also:
*******
$ diff bigip.pm bigip12.pm
1c1
< package bigip;
---
> package bigip12;
75c75
< $ENV{'TERM'} = "vt100-w";
---
> $ENV{'TERM'} = "vt100";
*******
And:
*******
$ diff f5rancid f5rancid-v9
64c64,65
< $ENV{'TERM'} = "vt100";
---
> #$ENV{'TERM'} = "vt100";
> $ENV{'TERM'} = "vt100-w";
186a188,191
> if (/^(\s*)community \S+/ && $filter_commstr) {
> ProcessHistory("SHOWBASE","","","# $1community <removed>\n");
> next;
> }
190a196,199
> if (/^(\s*)password crypt \S+/) {
> ProcessHistory("SHOWBASE","","","# $1password crypt <removed>\n");
> next;
> }
225c234,236
<
---
> if (/^(.*)\.password = / && $filter_pwds >= 1) {
> ProcessHistory("SHOWDB","","","# $1.password = <removed>\n") && next;
> }
269a281,286
> if (/^(\s*)monitor state (up|down)$/) {
> ProcessHistory("SHOWDB","","","# $1monitor state <removed>\n") && next;
> }
> if (/^(\s*)community \S+/ && $filter_commstr) {
> ProcessHistory("SHOWDB","","","# $1community <removed>\n") && next;
> }
277c294
< if (/^(\s*)password / && $filter_pwds >= 1) {
---
> if (/^(\s*)password (\w*)/ && $filter_pwds >= 1 && $2 ne "none") {
427a445,447
> return (1) if (/BIGpipe:.*: syntax error/);
> return (0) if ($found_end); # Only do this routine once
>
441a462,465
> if (/^(\s*)community \S+/ && $filter_commstr) {
> ProcessHistory("","","","# $1community <removed>\n") && next;
> }
>
532a557
> {'bigpipe list all' => 'WriteTerm'},
*******
Kind Regards,
Dan
On Mon, Sep 12, 2016 at 8:40 PM, Alan McKinnon <alan.mckinnon at gmail.com> wrote:
> Shaun,
>
> Usually setting PATH correctly is enough, but I don't think that is your
> problem. To check that I understand the problem correctly:
>
> All your devices work correctly except BIGIP 12. Even BIGIP 11 works
> properly. BIGIP 12 fails from a cronjob but works correctly from the command
> line as rancid user. This then excludes mistakes in router.db
>
> Correct?
>
> Nothing in cron or the environment strikes me as being unique to the BIGIP
> 12. For completeness, what is in the rancid user's environment in a regular
> login shell?
>
> If that doesn't reveal anything useful, I'd generate .raw files for both
> device types and compare them. Then read those files directly into rancid;
> at some point a difference in behaviour must show up
>
>
> On 12/09/2016 10:24, Shaun Krok wrote:
>>
>> Thanks Alan
>>
>> Always have been running as *user* = rancid
>> [rancid at Rancid bin]$ whoami
>> Rancid
>>
>> # Check every 30 mins
>> */30 * * * * /usr/local/rancid/bin/rancid-run &
>>
>> What do you suggest with regards setting up the cron environment for
>> rancid --- never had to prior BIGIP 12.0 and all other device types: Cisco,
>> Juniper, Riverbed , F5 working 100%
>>
>>
>>
>> -----Original Message-----
>> From: Rancid-discuss [mailto:rancid-discuss-bounces at shrubbery.net] On
>> Behalf Of Alan McKinnon
>> Sent: Monday, September 12, 2016 11:15 AM
>> To: rancid-discuss at shrubbery.net
>> Subject: Re: [rancid] F5 Rancid | BIGIP 12.0 | missed cmd(s)
>>
>> This problem almost always comes down to the usual issues with cron:
>>
>> - cron runs from a non-interactive non-login shell, meaning that the
>> environment is empty (the environment being nothing more than a handy
>> *user* convenience). Make sure your cron env is set up correctly and
>> don't run ./rancid-run, specify the full correct path
>>
>> - which user's cron is this in? It must go in rancid's crontab, not
>> root. If this is the case with your site, make sure you chown all
>> relevant data files back to the rancid user as some of them will now be
>> owned by root
>>
>>
>> On 12/09/2016 09:17, Shaun Krok wrote:
>>>
>>> Hi
>>>
>>> I have tried to look into forums where other users of rancid experience
>>> the same issue but could not find anything !
>>>
>>> Our issue as follows :
>>> We have been running rancid successfully against F5 LTM clusters (HA) for
>>> a few years now.
>>> The issue with partitions and TMSH was resolved and we have had no issues
>>> accept with the recent upgrade to BIGIP image : BIG-IP 12.0.0
>>>
>>> The issue is below.
>>> What we have noticed that if we use the cli command : ./rancid-run -r
>>> lon-f5-pri GLN & the collection has no issues, diff is done, entered into
>>> CVS and emails received !
>>>
>>> The problem is when the general cron runs and we receive the below in the
>>> logs.
>>> Using the NOPIPE I see the .raw file and have looked into the file and do
>>> not see any issues.
>>>
>>> The f5rancid script has been working as I said for years until version
>>> 12.0 of BIGIP.
>>> It works against several other production sites but running version 11.6
>>> of BIGIP
>>>
>>> I realize the below is just the process saying there is an issue but if
>>> anyone has suggestions would be much appreciated !
>>>
>>> Version of rancid : ## rancid 3.2.99
>>>
>>> ***************************************
>>> a snip of the crontab
>>> # Check every 30 mins
>>> */30 * * * * /usr/local/rancid/bin/rancid-run &
>>> **************************************
>>>
>>> **************************************
>>> A snip of the f5rancid script
>>> # tmsh commands, BIGIP v11
>>> @tmsh_commandtable = (
>>> {'tmsh show /sys version' => 'ShowVersion'},
>>> {'tmsh show /sys hardware' => 'ShowHardware'},
>>> {'tmsh show /sys license' => 'ShowLicense'},
>>> #{'cat /config/ZebOS.conf' => 'ShowZebOSconf'},
>>> #{'lsof -i :179' => 'ShowZebOSsockets'},
>>> {'tmsh show /net route static' => 'ShowRouteStatic'},
>>> #{'ls --full-time --color=never /config/ssl/ssl.crt' => 'ShowSslCrt'},
>>> #{'ls --full-time --color=never /config/ssl/ssl.key' => 'ShowSslKey'},
>>> #{'tmsh -q list' => 'WriteTermTMSH'},
>>> #{'tmsh -q -c /"cd /;list recursive"/' => 'WriteTermTMSH'},
>>> {'./f5part' => 'WriteTermTMSH'},
>>> **************************************
>>>
>>>
>>> Trying to get all of the configs.
>>> lon-f5-pri: missed cmd(s): tmsh show /net route static,tmsh show /sys
>>> hardware
>>> lon-f5-sec: missed cmd(s): tmsh show /net route static,tmsh show /sys
>>> version,tmsh show /sys hardware,tmsh show /sys license
>>> =====================================
>>> Getting missed routers: round 1.
>>> lon-f5-sec: missed cmd(s): tmsh show /net route static,tmsh show /sys
>>> version,tmsh show /sys hardware,tmsh show /sys license
>>> lon-f5-pri: missed cmd(s): tmsh show /net route static,tmsh show /sys
>>> hardware
>>> =====================================
>>> Getting missed routers: round 2.
>>> lon-f5-sec: missed cmd(s): tmsh show /net route static,tmsh show /sys
>>> version,tmsh show /sys hardware,tmsh show /sys license
>>> lon-f5-pri: missed cmd(s): tmsh show /net route static,tmsh show /sys
>>> hardware
>>> =====================================
>>> Getting missed routers: round 3.
>>> lon-f5-pri: missed cmd(s): tmsh show /net route static,tmsh show /sys
>>> hardware
>>> lon-f5-sec: missed cmd(s): tmsh show /net route static,tmsh show /sys
>>> version,tmsh show /sys hardware,tmsh show /sys license
>>> =====================================
>>> Getting missed routers: round 4.
>>> lon-f5-sec: missed cmd(s): tmsh show /net route static,tmsh show /sys
>>> version,tmsh show /sys hardware,tmsh show /sys license
>>> lon-f5-pri: missed cmd(s): tmsh show /net route static,tmsh show /sys
>>> hardware
>>>
>>>
>>> regards
>>>
>>>
>>>
>>> Shaun Krok
>>> Network Team
>>>
>>> Herzliya Business Park
>>> Herzliya Pituach 46140
>>> Israel
>>>
>>> Telephone: +972 (0)732889406
>>> Mobile: +972 (0)50 2424381
>>> email:shaun.krok at 888holdings.com
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> This email message and its attachments are for the sole use of the
>>> intended recipient(s) and may not be shared with any other party. They may
>>> contain confidential information of 888 Holdings plc or its direct and
>>> indirect subsidiaries (together, the “888 Group”) and are to be regarded as
>>> confidential information under any non-disclosure agreement. Any review,
>>> use, disclosure or distribution by persons or entities other than the
>>> intended recipient(s) is prohibited. Nothing in this message is capable of
>>> or intended to create any legally binding obligation. The 888 Group will
>>> only ever assume a legally binding obligation where recorded in a written
>>> agreement duly executed by the authorized signatories of the relevant 888
>>> Group company. The 888 Group accepts no liability for any personal views
>>> expressed in this message. If you are not the intended recipient, please
>>> contact the sender by return and destroy all copies of the original message
>>> and its attachments. Thank you
>>> _______________________________________________
>>> Rancid-discuss mailing list
>>> Rancid-discuss at shrubbery.net
>>> http://www.shrubbery.net/mailman/listinfo/rancid-discuss
>>>
>>
>> _______________________________________________
>> Rancid-discuss mailing list
>> Rancid-discuss at shrubbery.net
>> http://www.shrubbery.net/mailman/listinfo/rancid-discuss
>>
>>
>> This email message and its attachments are for the sole use of the
>> intended recipient(s) and may not be shared with any other party. They may
>> contain confidential information of 888 Holdings plc or its direct and
>> indirect subsidiaries (together, the “888 Group”) and are to be regarded as
>> confidential information under any non-disclosure agreement. Any review,
>> use, disclosure or distribution by persons or entities other than the
>> intended recipient(s) is prohibited. Nothing in this message is capable of
>> or intended to create any legally binding obligation. The 888 Group will
>> only ever assume a legally binding obligation where recorded in a written
>> agreement duly executed by the authorized signatories of the relevant 888
>> Group company. The 888 Group accepts no liability for any personal views
>> expressed in this message. If you are not the intended recipient, please
>> contact the sender by return and destroy all copies of the original message
>> and its attachments. Thank you
>>
>
> _______________________________________________
> Rancid-discuss mailing list
> Rancid-discuss at shrubbery.net
> http://www.shrubbery.net/mailman/listinfo/rancid-discuss
More information about the Rancid-discuss
mailing list