[rancid] control_rancid slow start

[heasley]

Mon, Nov 10, 2014 at 01:10:15AM -0500, Robert Drake:
> has anyone looked at the control_rancid script recently?   Here are some 
> timestamps from an example run from me (with a couple of added date 
> stamps to show where the time goes).  I'm only running rancid against 
> one file, but there are 1400 total devices in the group.
> 
> rancid-run -r <gw1-test-node> <testgroup>
> 
> starting: Mon Nov 10 00:33:34 EST 2014
> begin control_rancid: Mon Nov 10 00:33:34 EST 2014
> 
> Trying to get all of the configs. Mon Nov 10 00:35:33 EST 2014

it takes ~3s for 200 devices in svn.  i haven't timed cvs yet.  is it possible
that you have defined cvswrappers that are slow?  or you have a massive cvs
history file that is slowing making the operation slow?

> All routers sucessfully completed.
> 
> cvs diff: Diffing .
> cvs diff: Diffing configs
> cvs commit: Examining .
> cvs commit: Examining configs
> 
> ending: Mon Nov 10 00:35:37 EST 2014
> 
> 
> If I comment the following code out it runs in less than 3 seconds:
> 
> # check for 'up' routers missing in RCS.  no idea how this happens to 
> some folks
> for router in `cut -d\; -f1 ../routers.up` ; do
>      if [ $RCSSYS = cvs ] ; then
>      cvs status $router | grep -i 'status: unknown' > /dev/null 2>&1
>      else
>      svn status $router | grep '^?' > /dev/null 2>&1
>      fi
>      if [ $? -eq 0 ] ; then
>      touch $router
>      if [ $RCSSYS = cvs ] ; then
>          cvs add -ko $router
>      else
>          svn add $router
>      fi
>      echo "$RCSSYS added missing router $router"
>      fi
> done
> 
> Possible better option would be this (I think this will work with svn 
> but I don't have a tree to test it on):
> 
>   cut -d: -f1 ../routers.up | xargs cvs status | grep -i 'status: unknown'
> 
> Example test case:
> 
>   (echo test ; cut -d: -f1 ../routers.up) | xargs cvs status | grep -i 
> 'status: unknown'
> cvs status: nothing known about test
> File: no file test              Status: Unknown

that doesnt quite work for non-existent files.

> Another option might be to have a CLI argument that says "skip 
> rebuilding router.db.* and checking CVS stuff because we're reasonably 
> certain that is fine right now".  Finally, I would recommend abstracting 
> most of the router.db.* rebuild into another script and rewriting it in 
> perl because it's almost unreadable now.

i can see treating -r differently, but still maintain the integrity check.

> I can submit patches if this is too much for informal email.
> 
> Thanks,
> Robert
> _______________________________________________
> Rancid-discuss mailing list
> Rancid-discuss at shrubbery.net
> http://www.shrubbery.net/mailman/listinfo/rancid-discuss