Questions & Answers

pnda and swift errors

0 votes
asked Jul 30, 2017 by Saqib Arfeen (560 points)
Hello
I am trying to install pnda on openstack. After resolving a lot of problem , the point that I am stuck in is that swift is not working for pnda. I search it a lot and found that it may be swift tempurl and I enabled it in proxy-server.conf like this:-
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk ratelimit tempurl authtoken keystoneauth container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server

But still when I run ./heat_cli.py -f pico -e cation -s cation , after running for an hour it gives me error like this one:-

2017-07-29 15:07:16Z [saltmaster_server]: CREATE_IN_PROGRESS  state changed
2017-07-29 15:07:17Z [router_interface]: CREATE_COMPLETE  state changed
2017-07-29 15:07:26Z [saltmaster_server]: CREATE_COMPLETE  state changed
2017-07-29 15:07:26Z [deploy_package]: CREATE_IN_PROGRESS  state changed
2017-07-29 16:06:58Z [deploy_package]: CREATE_FAILED  RemoteError: resources.deploy_package: Remote error: ClientException Object HEAD failed: http://172.18.1.9:8080/v1/AUTH_bee802789f5b4a41b10e527d8caba478/2d2626b6-7ee0-4b66-8f01-4b04a8b9f22d/cation29072017night-deploy_package-bn6t2gzd4tg6 401 Unauthorized

2017-07-29 16:06:58Z [cation29072017night]: CREATE_FAILED  Resource CREATE failed: RemoteError: resources.deploy_package: Remote error: ClientException Object HEAD failed: http://172.18.1.9:8080/v1/AUTH_bee802789f5b4a41b10e527d8caba478/2d2626b6-7ee0-4b66-8f01-4b04a8b9f22d/cation29072017night-deploy_package-bn6t2g
Stack cation29072017night CREATE_FAILED
Command 'openstack stack create --timeout 1200 --wait --template pnda.yaml --environment pnda_env.yaml --parameter ZookeeperNodes=0 --parameter KafkaNodes=1 --parameter DataNodes=1 --parameter OpentsdbNodes=0 --parameter PndaFlavor=pico --parameter KeyName=cation29072017night cation29072017night' failed

I have been stuck there for a while and cannot proceed furthur. Any help or support is highly appreciated!
Saqib

1 Answer

0 votes
answered Jul 30, 2017 by trsmith2 (3,440 points)

Can you verify that the token expiration settings in /etc/keystone/keystone.conf are set reasonably for the length of time it takes you to create PNDA in your environment?

commented Jul 31, 2017 by Saqib Arfeen (560 points)
Thank you for quick response. I have increased the keystone token expiration time from 1 hour to 10 hours. The deploy process (i.e ./heat_cli -f pico -e cation) has been stuck in the " [deploy_package]: CREATE_IN_PROGRESS  state changed" step for about 7 hours, despite I have a local package server.Can I see real time debug logs of pnda, to see where it is stuck?
commented Jul 31, 2017 by trsmith2 (3,440 points)
Use this script to see if there were errors during the provisioning process - https://github.com/pndaproject/pnda-heat-templates/blob/develop/tools/check-stack.sh

For the SaltStack output, you are looking for Failed being non-zero.

If nothing shows up here, then you'll need to have a look in the /var/log on a node that is created (the saltmaster node is at the top of the nested stacks so you should at least see that node) and search for something uniquely related to your installation (such as the package server address you've configured). That should lead you to what is 'hanging' in the process. You should be able to get access to this node using the auto-generated key, which will be visible as one of the stack outputs (use openstack stack output list <stack> to see these). You may need to give it a floating IP address or create another instance attached to the same network to gain the ability to connect ssh.

That said, this is the bottom up approach. Top down you could check everything in your configuration - in particular make sure every address you've specified is actually reachable from the cluster under creation and also that the Internet is reachable - in both cases this must be either direct connection or via *transparent* proxy.
commented Aug 4, 2017 by Saqib Arfeen (560 points)
Thanks Sir , I just followed your advice to check the /var/log/messages and found a networking problem with my l3-agent and dhcp-agent HA . I fixed that and now it took much longer (about 10 hours) but finally deployed pnda heat stack.
But still I can see some errors in ./heat_cli logs -e cation.
I am stuck for a while on it and donot know where to go from here. (Should I destroy and redeploy the whole pnda stack or please tell me the way to only run saltstack .sls scripts for failed nodes ?)
https://pastebin.com/WZKuFYzX
...