landscape: agent-state: error details: hook failed ... why?

Question

I am attempting the AutopilotBeta deployment of OpenStack on 8 physical servers. I originally had an issue with a missing size attribute (thank you Dell for being ... different). But the latest version of openstack-install has addressed that issue. I was still having some odd authentication issues, so I took my environment back to bare metal and reinstalled from scratch. The MAAS install goes well, no issues. I am able to commission all of the machines. When I attempt to execute the "openstack-install", it makes it through the juju bootstrapping, but fails during the landscape deployment. I will put the pertinent log below. I stripped this environment down to bare metal twice with the same results ... so I am consistent. What am I doing wrong?

[DEBUG • 04-07 13:35:28] • cloudinstall.multi_install • (multi_install.py, do_install, 173)]
Finished MAAS step, now deploying Landscape.
[ERROR • 04-07 13:44:18] • cloudinstall.multi_install • (multi_install.py, run_deployer, 770)]
Problem deploying Landscape: {'output': '2015-04-07 13:35:44 
[DEBUG] deployer.cli: Using runtime GoEnvironment on maas\n2015-04-07 13:35:44 
[INFO] deployer.cli: Starting deployment of landscape-dense-maas\n2015-04-07 13:35:44 
[DEBUG] deployer.import: Getting charms...\n2015-04-07 13:35:44 
[DEBUG] deployer.charm: Cache dir /home/peugenis/.cloud-install/juju/.deployer-store-cache/cs_trusty_haproxy-4\n2015-04-07 13:35:44 
[DEBUG] deployer.charm: Retrieving store charm cs:trusty/haproxy-4\n2015-04-07 13:35:51 
[DEBUG] deployer.charm: Cache dir /home/peugenis/.cloud-install/juju/.deployer-store-cache/cs_trusty_rabbitmq-server-26\n2015-04-07 13:35:51 
[DEBUG] deployer.charm: Retrieving store charm cs:trusty/rabbitmq-server-26\n2015-04-07 13:35:57 
[DEBUG] deployer.charm: Cache dir /home/peugenis/.cloud-install/juju/.deployer-store-cache/cs_trusty_postgresql-15\n2015-04-07 13:35:57 
[DEBUG] deployer.charm: Retrieving store charm cs:trusty/postgresql-15\n2015-04-07 13:36:04 
[DEBUG] deployer.charm: Cache dir /home/peugenis/.cloud-install/juju/.deployer-store-cache/cs_trusty_landscape-server\n2015-04-07 13:36:09 
[DEBUG] deployer.charm: Retrieving store charm cs:trusty/landscape-server-8\n2015-04-07 13:36:15 
[DEBUG] deployer.charm: Cache dir /home/peugenis/.cloud-install/juju/.deployer-store-cache/cs_trusty_landscape-server\n2015-04-07 13:36:15 
[DEBUG] deployer.charm: Cache dir /home/peugenis/.cloud-install/juju/.deployer-store-cache/cs_trusty_apache2-10\n2015-04-07 13:36:15 
[DEBUG] deployer.charm: Retrieving store charm cs:trusty/apache2-10\n2015-04-07 13:36:22 
[DEBUG] deployer.deploy: Resolving configuration\n2015-04-07 13:36:22 
[DEBUG] deployer.env: Connecting to environment...\n2015-04-07 13:36:22 
[DEBUG] deployer.env: Connected to environment\n2015-04-07 13:36:22 
[INFO] deployer.import: Deploying services...\n2015-04-07 13:36:22 
[DEBUG] deployer.import: <deployer.env.go.GoEnvironment object at 0x7f32ce001c10>\n2015-04-07 13:36:23 
[INFO] deployer.import:  Deploying service apache2 using cs:trusty/apache2-10\n2015-04-07 13:36:23 
[DEBUG] deployer.import:  Refetching status for placement deploys\n2015-04-07 13:36:44 
[DEBUG] deployer.import:  Setting annotations\n2015-04-07 13:36:44 
[INFO] deployer.import:  Deploying service haproxy using cs:trusty/haproxy-4\n2015-04-07 13:37:00 
[DEBUG] deployer.import:  Setting annotations\n2015-04-07 13:37:00 
[INFO] deployer.import:  Deploying service landscape using cs:trusty/landscape-server\n2015-04-07 13:37:22 
[DEBUG] deployer.import:  Setting annotations\n2015-04-07 13:37:23 
[INFO] deployer.import:  Deploying service landscape-msg using cs:trusty/landscape-server\n2015-04-07 13:37:41 
[DEBUG] deployer.import:  Setting annotations\n2015-04-07 13:37:42 
[INFO] deployer.import:  Deploying service postgresql using cs:trusty/postgresql-15\n2015-04-07 13:37:58 
[DEBUG] deployer.import:  Setting annotations\n2015-04-07 13:37:58 
[INFO] deployer.import:  Deploying service rabbitmq-server using cs:trusty/rabbitmq-server-26\n2015-04-07 13:38:14 
[DEBUG] deployer.import:  Setting annotations\n2015-04-07 13:38:20 
[DEBUG] deployer.import: Adding units...\n2015-04-07 13:38:20 
[DEBUG] deployer.import:  Service \'apache2\' does not need any more units added.\n2015-04-07 13:38:20 
[DEBUG] deployer.import:  Service \'haproxy\' does not need any more units added.\n2015-04-07 13:38:20 
[DEBUG] deployer.import:  Service \'landscape\' does not need any more units added.\n2015-04-07 13:38:20 
[DEBUG] deployer.import:  Service \'landscape-msg\' does not need any more units added.\n2015-04-07 13:38:20 
[DEBUG] deployer.import:  Service \'postgresql\' does not need any more units added.\n2015-04-07 13:38:20 
[DEBUG] deployer.import:  Service \'rabbitmq-server\' does not need any more units added.\n2015-04-07 13:38:20 
[DEBUG] deployer.import: Waiting for units before adding relations\n2015-04-07 13:40:02 
[DEBUG] deployer.env:  Delta machine: 0/lxc/0 change:pending\n2015-04-07 13:40:12 
[DEBUG] deployer.env:  Delta machine: 0/lxc/1 change:pending\n2015-04-07 13:40:22 
[DEBUG] deployer.env:  Delta machine: 0/lxc/0 change:pending\n2015-04-07 13:40:22 
[DEBUG] deployer.env:  Delta machine: 0/lxc/0 change:started\n2015-04-07 13:40:22 
[DEBUG] deployer.env:  Delta unit: apache2/0 change:installing\n2015-04-07 13:40:22 
[DEBUG] deployer.env:  Delta unit: apache2/0 change:installing\n2015-04-07 13:40:22 
[DEBUG] deployer.env:  Delta machine: 0/lxc/2 change:pending\n2015-04-07 13:40:32 
[DEBUG] deployer.env:  Delta machine: 0/lxc/3 change:pending\n2015-04-07 13:40:47 
[DEBUG] deployer.env:  Delta machine: 0/lxc/4 change:pending\n2015-04-07 13:40:57 
[DEBUG] deployer.env:  Delta machine: 0/lxc/1 change:pending\n2015-04-07 13:40:57 
[DEBUG] deployer.env:  Delta machine: 0/lxc/1 change:started\n2015-04-07 13:40:57 
[DEBUG] deployer.env:  Delta machine: 0/lxc/2 change:pending\n2015-04-07 13:40:57 
[DEBUG] deployer.env:  Delta unit: haproxy/0 change:installing\n2015-04-07 13:40:57 
[DEBUG] deployer.env:  Delta machine: 0/lxc/2 change:started\n2015-04-07 13:40:57 
[DEBUG] deployer.env:  Delta unit: haproxy/0 change:installing\n2015-04-07 13:40:57 
[DEBUG] deployer.env:  Delta unit: landscape/0 change:installing\n2015-04-07 13:40:57 
[DEBUG] deployer.env:  Delta unit: landscape/0 change:installing\n2015-04-07 13:40:57 
[DEBUG] deployer.env:  Delta machine: 0/lxc/3 change:started\n2015-04-07 13:40:57 
[DEBUG] deployer.env:  Delta unit: landscape-msg/0 change:allocating\n2015-04-07 13:41:02 
[DEBUG] deployer.env:  Delta machine: 0/lxc/5 change:pending\n2015-04-07 13:41:02 
[DEBUG] deployer.env:  Delta unit: landscape-msg/0 change:installing\n2015-04-07 13:41:02 
[DEBUG] deployer.env:  Delta unit: landscape-msg/0 change:installing\n2015-04-07 13:41:12 
[DEBUG] deployer.env:  Delta machine: 0/lxc/4 change:pending\n2015-04-07 13:41:12 
[DEBUG] deployer.env:  Delta machine: 0/lxc/4 change:started\n2015-04-07 13:41:13 
[DEBUG] deployer.env:  Delta unit: postgresql/0 change:installing\n2015-04-07 13:41:13 
[DEBUG] deployer.env:  Delta unit: apache2/0 change:installing\n2015-04-07 13:41:18 
[DEBUG] deployer.env:  Delta unit: haproxy/0 change:installing\n2015-04-07 13:41:23 
[DEBUG] deployer.env:  Delta machine: 0/lxc/5 change:pending\n2015-04-07 13:41:23 
[DEBUG] deployer.env:  Delta machine: 0/lxc/5 change:started\n2015-04-07 13:41:23 
[DEBUG] deployer.env:  Delta unit: rabbitmq-server/0 change:allocating\n2015-04-07 13:41:28 
[DEBUG] deployer.env:  Delta unit: apache2/0 change:started\n2015-04-07 13:41:28 
[DEBUG] deployer.env:  Delta unit: rabbitmq-server/0 change:installing\n2015-04-07 13:41:28 
[DEBUG] deployer.env:  Delta unit: rabbitmq-server/0 change:installing\n2015-04-07 13:41:38 
[DEBUG] deployer.env:  Delta unit: haproxy/0 change:started\n2015-04-07 13:41:58 
[DEBUG] deployer.env:  Delta unit: postgresql/0 change:installing\n2015-04-07 13:41:58 
[DEBUG] deployer.env:  Delta unit: postgresql/0 change:installing\n2015-04-07 13:42:08 
[DEBUG] deployer.env:  Delta unit: postgresql/0 change:started\n2015-04-07 13:42:43 
[DEBUG] deployer.env:  Delta unit: rabbitmq-server/0 change:installing\n2015-04-07 13:44:18 
[DEBUG] deployer.env:  Delta unit: landscape-msg/0 change:error\n2015-04-07 13:44:18 
[ERROR] deployer.env: The following units had errors:\n   unit: landscape-msg/0: machine: 0/lxc/3 agent-state: error details: hook failed: "install"\n2015-04-07 13:44:18 
[INFO] deployer.cli: Deployment stopped. run time: 513.42\n', 'status': 1}
[DEBUG • 04-07 13:44:18] • cloudinstall.utils • (utils.py, global_exchandler, 63)]
Traceback (most recent call last):
 File "/usr/share/openstack/cloudinstall/utils.py", line 78, in run
super().run()
 File "/usr/lib/python3.4/threading.py", line 868, in run
self._target(*self._args, **self._kwargs)
 File "/usr/share/openstack/cloudinstall/machinewait.py", line 127, in do_continue
 self.installer.do_install()
 File "/usr/share/openstack/cloudinstall/multi_install.py", line 177, in do_install
self.loop).run()
File "/usr/share/openstack/cloudinstall/multi_install.py", line 708, in run
self.deploy_landscape()
File "/usr/share/openstack/cloudinstall/multi_install.py", line 731, in deploy_landscape
self.run_deployer()
  File "/usr/share/openstack/cloudinstall/multi_install.py", line 771, in   run_deployer
raise Exception("Error deploying Landscape.")
Exception: Error deploying Landscape.

Additional information. I am seeing the following errors in the landscape log on the machine being deployed to:

2015-04-07 20:44:29 INFO unit.landscape/0.install logger.go:40 
2015-04-07 20:44:29 INFO unit.landscape/0.install logger.go:40 E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/z/zope.browserresource/python-zope.browserresource_3.12.0-0ubuntu1_all.deb  503  Service Unavailable

But I am able to wget that file from that machine.

Two lines from the /var/log/maas/proxy/store.log that relate to the above:

1428439419.359 RELEASE -1 FFFFFFFF 3C1B6A761DD7EF328EC4C7C7822C849A  503 1428439419         0        -1 text/html 3571/3571 GET http://archive.ubuntu.com/ubuntu/pool/universe/z/zope.browserresource/python-zope.browserresource_3.12.0-0ubuntu1_all.deb
1428439419.367 RELEASE -1 FFFFFFFF 46EC83BBA3477FFD0A0EE9199555EA91  503 1428439419         0        -1 text/html 3571/3571 GET http://archive.ubuntu.com/ubuntu/pool/universe/z/zope.browserresource/python-zope.browserresource_3.12.0-0ubuntu1_all.deb

As suggested, I checked the proxy log on the MAAS server and it is reporting 503 for the files that are being reported as failing in the unit-landscape-msg-0.log file. Well that is a bit embarrassing, but now I know about another set of logs to check. Is there a way to just redeploy the landscape container alone or am I better off doing the "sudo openstack-install -u" followed by another "sudo openstack-install"? — Pat E, Apr 08 '15 at 01:10

score 2 · Accepted Answer · answered Apr 08 '15 at 00:14

Lots of units deployed just fine according to your log, just landscape-msg/0 failed:

[ERROR] deployer.env: The following units had errors:\n unit: landscape-msg/0: machine: 0/lxc/3 agent-state: error details: hook failed: "install"

It would be very helpful if you could get the unit-landscape-msg-0.log file from /var/log/juju in the bootstrap node.

You can also check the proxy logs on the maas server at /var/log/maas/proxy/ and see if there was some random failure while packages were being fetched via apt-get.

Sure enough ... transient (albeit consistent ... for about an hour) ... error. Thank you for showing me where those other logs are. Knowing where they are will answer a lot of my questions before I have to post here. — Pat E, Apr 08 '15 at 02:24

landscape: agent-state: error details: hook failed ... why?

1 Answers1