Franck
2015-06-16 02:12:03 UTC
We've been experiencing a lot of "Command exceeded timeouts" on basic shell
commands using the "exec" type for tasks that should execute fairly fast:
Jun 15 15:45:44 host1 puppet-agent[57648]:
(/Stage[main]/Timezone::Utc/Exec[/bin/rm -f /etc/localtime && /bin/ln -s
/usr/share/zoneinfo/UTC /etc/localtime]) Command exceeded timeout
Jun 10 21:15:24 host1 puppet-agent[57081]:
(/Stage[main]/Open-vm-tools::Package/Exec[/usr/bin/vmware-uninstall-tools.pl]/onlyif)
Check "/usr/bin/test -f /usr/bin/vmware-uninstall-tools.pl" exceeded timeout
Jun 10 23:56:02 host1 puppet-agent[40286]:
(/Stage[main]/Open-vm-tools::Package/Exec[/usr/bin/yum install -y
open-vm-tools.x86_64]/unless) Check "/bin/rpm -q open-vm-tools" exceeded
timeout
All these commands can be run locally to the host and return fairly
quickly, but when puppet executes them they time out. Extending the timeout
is an option but ridiculous since default is 300 seconds and none of these
commands should take 5 minutes or more to return.
Some of the things observed is that this only affects CentOS 6.x hosts as
we also have Ubuntu 14.x hosts and they do not experience these problems.
Also, we've played around with different versions of the puppet agent
along with different versions of Ruby and none of them had any effect as
this condition persists regardless. Also, this does not seem to affect all
of our CentOS 6.x hosts but only certain ones -- randomly. Running puppet
agent in debug mode does not seem to uncover what's going on as it just
hangs when it gets to the "exec".
It's very annoying and actually dangerous in some cases as the puppet agent
will continue spawning multiple "applying configuration" processes which
will cause hosts to swap memory as each takes up more and more memory and
in some instances will hose them entirely.
We've had to remove these manifests that cause these conditions in the
interim but right now we have a lot of hosts we need to manage with puppet
so we need to be able to use this.
Basic info on the hosts in question:
- Puppet: 3.7.5
- Ruby: 2.1.2
- CentOS 6.6
Anyone have any ideas as to what could be causing this?
Thanks.
Franck
commands using the "exec" type for tasks that should execute fairly fast:
Jun 15 15:45:44 host1 puppet-agent[57648]:
(/Stage[main]/Timezone::Utc/Exec[/bin/rm -f /etc/localtime && /bin/ln -s
/usr/share/zoneinfo/UTC /etc/localtime]) Command exceeded timeout
Jun 10 21:15:24 host1 puppet-agent[57081]:
(/Stage[main]/Open-vm-tools::Package/Exec[/usr/bin/vmware-uninstall-tools.pl]/onlyif)
Check "/usr/bin/test -f /usr/bin/vmware-uninstall-tools.pl" exceeded timeout
Jun 10 23:56:02 host1 puppet-agent[40286]:
(/Stage[main]/Open-vm-tools::Package/Exec[/usr/bin/yum install -y
open-vm-tools.x86_64]/unless) Check "/bin/rpm -q open-vm-tools" exceeded
timeout
All these commands can be run locally to the host and return fairly
quickly, but when puppet executes them they time out. Extending the timeout
is an option but ridiculous since default is 300 seconds and none of these
commands should take 5 minutes or more to return.
Some of the things observed is that this only affects CentOS 6.x hosts as
we also have Ubuntu 14.x hosts and they do not experience these problems.
Also, we've played around with different versions of the puppet agent
along with different versions of Ruby and none of them had any effect as
this condition persists regardless. Also, this does not seem to affect all
of our CentOS 6.x hosts but only certain ones -- randomly. Running puppet
agent in debug mode does not seem to uncover what's going on as it just
hangs when it gets to the "exec".
It's very annoying and actually dangerous in some cases as the puppet agent
will continue spawning multiple "applying configuration" processes which
will cause hosts to swap memory as each takes up more and more memory and
in some instances will hose them entirely.
We've had to remove these manifests that cause these conditions in the
interim but right now we have a lot of hosts we need to manage with puppet
so we need to be able to use this.
Basic info on the hosts in question:
- Puppet: 3.7.5
- Ruby: 2.1.2
- CentOS 6.6
Anyone have any ideas as to what could be causing this?
Thanks.
Franck
--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f2071040-32e1-4c9b-8656-5a33650c67ca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+***@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f2071040-32e1-4c9b-8656-5a33650c67ca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.