2

I setup Ubuntu Openstack using Autopilot Installer / MAAS. When I try booting the VM the cloud-init fails to get metadata. Here's the Instance log (partial)

cloud-init-nonet[25.04]: static networking is now up
 * Starting configure network device[74G[ OK ]
Cloud-init v. 0.7.5 running 'init' at Fri, 20 Mar 2015 16:43:33 +0000. Up 25.26 seconds.
ci-info: ++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++
ci-info: +--------+------+--------------+---------------+-------------------+
ci-info: | Device |  Up  |   Address    |      Mask     |     Hw-Address    |
ci-info: +--------+------+--------------+---------------+-------------------+
ci-info: |   lo   | True |  127.0.0.1   |   255.0.0.0   |         .         |
ci-info: |  eth0  | True | 192.168.1.21 | 255.255.255.0 | fa:16:3e:ab:ac:16 |
ci-info: +--------+------+--------------+---------------+-------------------+
ci-info: +++++++++++++++++++++++++++++Route info++++++++++++++++++++++++++++++
ci-info: +-------+-------------+---------+---------------+-----------+-------+
ci-info: | Route | Destination | Gateway |    Genmask    | Interface | Flags |
ci-info: +-------+-------------+---------+---------------+-----------+-------+
ci-info: |   0   | 192.168.1.0 | 0.0.0.0 | 255.255.255.0 |    eth0   |   U   |
ci-info: +-------+-------------+---------+---------------+-----------+-------+
2015-03-20 16:43:33,409 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:43:34,422 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [1/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:43:35,431 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [2/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:43:36,435 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [3/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:43:37,440 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [4/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:43:38,444 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [5/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:43:40,449 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [7/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:43:42,455 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [9/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:43:44,460 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [11/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:43:46,466 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [13/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:43:48,472 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [15/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:43:51,484 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [18/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:43:54,491 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [21/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:43:57,503 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [24/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:44:00,520 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [27/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:44:03,541 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [30/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:44:07,556 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [34/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:44:11,569 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [38/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:44:15,580 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [42/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:44:19,594 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [46/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:44:23,607 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:44:28,622 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [55/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:44:33,641 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [60/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:44:38,670 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [65/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:44:43,690 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [70/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:44:48,710 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [75/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:44:54,726 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [81/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:45:00,741 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [87/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:45:06,757 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [93/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:45:12,770 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [99/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:45:18,790 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [105/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:45:25,805 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [112/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:45:32,830 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by : [Errno 101] Network is unreachable)]
2015-03-20 16:45:39,843 - DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 126 seconds
2015-03-20 16:45:39,855 - url_helper.py[WARNING]: Calling 'http://192.168.1.20//latest/meta-data/instance-id' failed [0/120s]: request error [HTTPConnectionPool(host='192.168.1.20', port=80): Max retries exceeded with url: //latest/meta-data/instance-id (Caused by : [Errno 111] Connection refused)]
2015-03-20 16:45:40,865 - url_helper.py[WARNING]: Calling 'http://192.168.1.20//latest/meta-data/instance-id' failed [1/120s]: request error [HTTPConnectionPool(host='192.168.1.20', port=80): Max retries exceeded with url: //latest/meta-data/instance-id (Caused by : [Errno 111] Connection refused)]
2015-03-20 16:45:41,879 - url_helper.py[WARNING]: Calling 'http://192.168.1.20//latest/meta-data/instance-id' failed [2/120s]: request error [HTTPConnectionPool(host='192.168.1.20', port=80): Max retries exceeded with url: //latest/meta-data/instance-id (Caused by : [Errno 111] Connection refused)]
2015-03-20 16:45:42,884 - url_helper.py[WARNING]: Calling 'http://192.168.1.20//latest/meta-data/instance-id' failed [3/120s]: request error [HTTPConnectionPool(host='192.168.1.20', port=80): Max retries exceeded with url: //latest/meta-data/instance-id (Caused by : [Errno 111] Connection refused)]
2015-03-20 16:45:43,889 - url_helper.py[WARNING]: Calling 'http://192.168.1.20//latest/meta-data/instance-id' failed [4/120s]: request error [HTTPConnectionPool(host='192.168.1.20', port=80): Max retries exceeded with url: //latest/meta-data/instance-id (Caused by : [Errno 111] Connection refused)]
2015-03-20 16:45:44,894 - url_helper.py[WARNING]: Calling 'http://192.168.1.20//latest/meta-data/instance-id' failed [5/120s]: request error [HTTPConnectionPool(host='192.168.1.20', port=80): Max retries exceeded with url: //latest/meta-data/instance-id (Caused by : [Errno 111] Connection refused)]
2015-03-20 16:45:46,900 - url_helper.py[WARNING]: Calling 'http://192.168.1.20//latest/meta-data/instance-id' failed [7/120s]: request error [HTTPConnectionPool(host='192.168.1.20', port=80): Max retries exceeded with url: //latest/meta-data/instance-id (Caused by : [Errno 111] Connection refused)]
2015-03-20 16:45:48,912 - url_helper.py[WARNING]: Calling 'http://192.168.1.20//latest/meta-data/instance-id' failed [9/120s]: request error [HTTPConnectionPool(host='192.168.1.20', port=80): Max retries exceeded with url: //latest/meta-data/instance-id (Caused by : [Errno 111] Connection refused)]
2015-03-20 16:45:50,918 - url_helper.py[WARNING]: Calling 'http://192.168.1.20//latest/meta-data/instance-id' failed [11/120s]: request error [HTTPConnectionPool(host='192.168.1.20', port=80): Max retries exceeded with url: //latest/meta-data/instance-id (Caused by : [Errno 111] Connection refused)]
2015-03-20 16:45:52,924 - url_helper.py[WARNING]: Calling 'http://192.168.1.20//latest/meta-data/instance-id' failed [13/120s]: request error [HTTPConnectionPool(host='192.168.1.20', port=80): Max retries exceeded with url: //latest/meta-data/instance-id (Caused by : [Errno 111] Connection refused)]
2015-03-20 16:45:54,930 - url_helper.py[WARNING]: Calling 'http://192.168.1.20//latest/meta-data/instance-id' failed [15/120s]: request error [HTTPConnectionPool(host='192.168.1.20', port=80): Max retries exceeded with url: //latest/meta-data/instance-id (Caused by : [Errno 111] Connection refused)]
2015-03-20 16:45:57,943 - url_helper.py[WARNING]: Calling 'http://192.168.1.20//latest/meta-data/instance-id' failed [18/120s]: request error [HTTPConnectionPool(host='192.168.1.20', port=80): Max retries exceeded with url: //latest/meta-data/instance-id (Caused by : [Errno 111] Connection refused)]
2015-03-20 16:46:00,956 - url_helper.py[WARNING]: Calling 'http://192.168.1.20//latest/meta-data/instance-id' failed [21/120s]: request error [HTTPConnectionPool(host='192.168.1.20', port=80): Max retries exceeded with url: //latest/meta-data/instance-id (Caused by : [Errno 111] Connection refused)]
2015-03-20 16:46:03,970 - url_helper.py[WARNING]: Calling 'http://192.168.1.20//latest/meta-data/instance-id' failed [24/120s]: request error [HTTPConnectionPool(host='192.168.1.20', port=80): Max retries exceeded with url: //latest/meta-data/instance-id (Caused by : [Errno 111] Connection refused)]

its also interesting to note that the IP went from 169.254.169.254 to 192.168.1.20 (192.x is the DHCP agent IP).

Any ideas where I can start?

0xF2
  • 3,154
Chirayu Patel
  • 61
  • Is this better Jason? -- http://pastebin.com/U9rt2k0b – Chirayu Patel Mar 20 '15 at 16:59
  • It seems like you have some network configuration problem with the VM. VM route table has only one configured route and no default gateway. It is for the 192.168.1.0/24 network. Later the .py script tries to access IP from other network (169.254.169.254). Without routes' configuration information VM networking does not know where to send the network packets. – zloster Mar 20 '15 at 19:07
  • @zloster Networking was configured as a part of Auto Pilot. – Chirayu Patel Mar 23 '15 at 16:23
  • 169.254/16 network is assigned for dynamically configured link-local addresses. They are only valid on a single network segment and are not to be routed. The network card/driver will self-assign such an address if there is no DHCP response. This question has some info. Also have you checked this question about MAAS/Canonica OpenStack Autopilot? – zloster Mar 23 '15 at 21:21

4 Answers4

1

You need to make sure your cluster controller is configured correctly so that the machine can actually access the region controller metadata server. You can do this with:

sudo dpkg-reconfigure maas-cluster-controller

And you need to input the IP address of the region controller in the form of:

http://<ip-address>/MAAS
Andres Rodriguez
  • 11
1

In my case (new install using Ubuntu Juju) the problem was that neutron was misconfigured with the wrong ip address for the metadata server.

This is good: http://techbackground.blogspot.ie/2013/06/metadata-via-dhcp-namespace.html

That pointed me at: 

root@neutron:/etc/neutron# cat metadata_agent.ini 
...
nova_metadata_ip = 192.168.100.151
nova_metadata_port = 8775
...

I fixed the nova_metadata_ip and restarted the neutron metadata service and that solved the issue.

bladedoyle
  • 13
  • 4
  • In openstack me also faced same problem, I followed below steps, It got worked for me. Thanks @bladedoyle.

    Navigate to your controller node and gave the following command sudo netstat -lntp | grep 8775

    Output: tcp 0 0 0.0.0.0:8775 0.0.0.0:* LISTEN 11166/python Navigate to this location -> "nano /etc/neutron/metadata_agent.ini" Update following values [DEFAULT] ... nova_metadata_ip = Ex: nova_metadata_ip = 192.168.1.10 nova_metadata_port = 8775

     – Shankar Oct 13 '16 at 09:48
0

I had a similar issue on my Vultr VPS. After debugging, I found I couldn't ping any IPv4 address from my VPS. The policy of iptables for INPUT is DROP. After running the following command, the ping function works normally and cloud-init runs correctly

iptables -I INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

Details can be found here https://serverfault.com/questions/356282/cannot-ping-outside-network-with-these-ip-rules

My suggestion of debugging this issue is to first use ip route show | grep 169.254.169.254 to get the gateway address. Then check the connection to gateway. If it's not reachable, try disabling firewall.

mach6
  • 101
0

I had a similar issue with an 8-node deployment (3 controllers, 1 logging, 4 computes) where any instances created on the tenant network could not be logged into b/c they did not have the necessary SSH key installed via the metadata agent service. The nova console-log showed failed attempts trying to reach the metadata agent service on the tenant network neutron router gateway (192.168.227.1). If I logged into the neutron router and arping'd the instance, it would start to work.

Performing a packet capture on the compute node, I found the instance sending ARP requests for its Neutron gateway 192.168.227.1. The MAC returned was not a Neutron MAC, but rather a Cisco one. Because traffic is getting routed to that Cisco device (where no metadata service exists), requests to the metadata address failed.

The proposed solution was to confirm the possible existence of an SVI with 192.168.227.1 configured on it, and remove it since the address should be configured on the Neutron router.

While editing the metadata_agent.ini file is a work-around, I would strongly caution using that as a 'fix' since those values are pulled from the openstack_user_config.yml file. If set correctly there, it should work as configured and with the default values in the .ini file. I changed the values in my .ini file to test the external endpoint, which resulted in instances I could log into, but this did not address the root problem of a duplicate tenant gateway IP, and what ulitmately needed to be corrected.

Thomas
  • 1