Leaving Barbossa would betray the fact that he doesn't have Will's full loyalty, before Will's finished working out how he's going to get the Pearl for himself. On the other hand, maybe Will can prove himself trustworthy by getting this done fast. After all, Barbossa already thinks him simple. Simple and trustworthy would be ideal.
Leaving Elizabeth would mean he didn't have to wait for her to talk to him.
"The temple's on a river. Two men can take the boat up there without much difficulty."
Re: Below deck
Leaving Barbossa would betray the fact that he doesn't have Will's full loyalty, before Will's finished working out how he's going to get the Pearl for himself. On the other hand, maybe Will can prove himself trustworthy by getting this done fast. After all, Barbossa already thinks him simple. Simple and trustworthy would be ideal.
Leaving Elizabeth would mean he didn't have to wait for her to talk to him.
"The temple's on a river. Two men can take the boat up there without much difficulty."
That's a 'yes'.