[SUMMARY] Cannot perform NFS mounts
2007-12-25 9:52:00
NFS mount directories from one of our servers, getting the dreaded
"permission denied" error. Everyone pointed to this being a problem
with the server and not the client.
Basically, the problem went away when I edited the /etc/hosts file on
the server and commented OUT the entries for the remote systems.
Can someone PLEASE explain to me why this is so? We are using DNS,
with the nameservers specified in /etc/resolv.conf. In /etc/nsswitch.conf
the entry for hosts is: "files dns".
Here are some other good suggestions I received:
1. Make sure the server is sharing the directories appropriately.
Check syntax of /etc/dfs/dfstab.
2. Could be NIS-related or DNS-related, caused by a number of
different configuration errors.
3. Check output of: "showmount -e <server>" to be sure the
directories are available for mounting.
4. One suggested that NOT having the IP/hostname in the /etc/hosts
file of the remote system can cause this behavior. I believe we
were using NFS mounts before these were entered into /etc/hosts
of the clients, so maybe this is a local configuration issue.
If anyone can shed some light on this please feel free!
5. One suggested that the systems might not "trust" each-other,
via the /etc/hosts.equiv and/or /.rhosts files. Again, I didn't
think these would be related to NFS mounts.
6. One suggested using snoop to look at packets, and check that
the remote system has the name right (FQDN vs simple name vs bad
/etc/hosts entry).
7. One person had experienced this before with an optical Jukebox,
and the problem was that the /usr/lib/nfs/mountd had to be
stopped, and then replaced by a new daemon supplied with the
Jukebox. This doesn't apply in my case, but it's useful info.
8. Check that client machine is able to resolve server name.
Put an entry in /etc/hosts for NFS server. This is what I
do not understand, BECAUSE THIS IS WHAT BROKE NFS HERE IN
THE FIRST PLACE.
9. One person suggested using the automounter, but we already
do for other directories and this was broken as well.
10. If using netgroups to restrict the shares, make sure that the
client did not get out of the netgroup.
11. Truss the mountd daemon on the server to see why it's it's
denying permission to the client (I haven't tried this yet).
BTW, someone was kind enough to send me the fix for the
"NetISAM: read error" message in /var/adm/messages. Thanks!
The problem is that the files in /TT_DB are corrupt on
the host . As root:
Suspend the rpc.ttdbserver program with:
kill -STOP <pid-#-of-rpc.ttdbserverd>
Remove all of the TT_DB directories and contents.
To get a list of all possible locations where the TT_DB directories
can exist for a host:
df -kF ufs | awk '{if (NR>1) print $6 "/TT_DB"}'
Remove them (back-quotes around the df command):
rm -rf `df -kF ufs | awk '{if (NR>1) print $6 "/TT_DB"}'`
Kill the rpc.ttdbserverd program with:
kill -KILL <pid-#-of-rpc.ttdbserverd>
The next request to rpc.ttdbserverd will restart the
program and re-create the */TT_DB files as they are
needed.
Thanks to:
Joel Lee <jlee@thomas.com>
Michael Hill <Michael_Hill@csgsystems.com>
Ross Helfand <rhelfand@Census.GOV>
Ron Kelley <rkelly@InfoAve.Net>
"Michael J. Connolly" <mjconnly@newton41.ckcorp.com>
Kevin.Sheehan@uniq.com.au (Kevin Sheehan {Consulting Poster Child})
Dennis Martens <MARTENSD@health.qld.gov.au>
Parthasarathi Palanisami <sarathip@cisco.com>
Ramindur Singh <ramindur@zuno.com>
Gustavo Chaves <gustavo@cpqd.com.br>
> System: Sparc5, Solaris 2.5
>
> Remote System: Sparc5, Solaris 2.5
>
> Problem:
>
> Cannot perform NFS mounts, whether listed in /etc/vfstab or
> enabled using the automounter; we get the message "Permission denied".
>
> This appears to be a problem with the local system, as the remote
> system is sharing the volumes appropriately, and we have no problem
> mounting them from other systems.
>
> I'm not sure if it could be related, but on this same system I had
> to remove all of the TT_DB tool-talk databases as one had become
> corrupted (got that "rpc.ttdbserverd[15368]: NetISAM: read failed"
> message).
>
> What follows is the truss output from the mount command which
> attempts to mount a directory that other systems have no problem
> mounting. I've put asterisks around the entries that look
> significant to me. What is strange is that mount tries to open the
> library file /usr/platform/SUNW,SPARCstation-5/lib/libc_psr.so.1
> which doesn't exist anywhere on ANY of our systems!
>
> This is the entry for /opt/AEgs in /etc/vfstab:
>
> galen:/opt/AEgs - /opt/AEgs nfs no yes hard,intr,ro,noquota
>
> If anyone has any ideas I would LOVE to hear them; this is a
> critical machine and of course access to the remote system is
> also critical. I'd rather not have to do a re-install of the OS.
>
> Thanks!
>
> Dave (and yes I will summarize!)
>
>
> # truss mount -F nfs -o hard,ro galen:/opt/AEgs /opt/AEgs
> execve("/usr/sbin/mount", 0xEFFFF390, 0xEFFFF3B0) argc = 7
> open("/dev/zero", O_RDONLY) = 3
> mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE, 3, 0) =
> 0xEF7C0000
> open("/usr/lib/libintl.so.1", O_RDONLY) = 4
> fstat(4, 0xEFFFF04C) = 0
> mmap(0x00000000, 4096, PROT_READ|PROT_EXEC, MAP_SHARED, 4, 0) = 0xEF7B0000
> mmap(0x00000000, 81920, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF790000
> munmap(0xEF793000, 61440) = 0
> mmap(0xEF7A2000, 2416, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4,
> 8192) = 0xEF7A2000
> mmap(0xEF7A3000, 920, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3,
> 0) = 0xEF7A3000
> close(4) = 0
> open("/usr/lib/libc.so.1", O_RDONLY) = 4
> fstat(4, 0xEFFFF04C) = 0
> mmap(0xEF7B0000, 4096, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) =
> 0xEF7B0000
> mmap(0x00000000, 618496, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF6C0000
> munmap(0xEF73E000, 61440) = 0
> mmap(0xEF74D000, 28768, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED,
> 4, 512000) = 0xEF74D000
> mmap(0xEF755000, 4792, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3,
> 0) = 0xEF755000
> close(4) = 0
> open("/usr/lib/libw.so.1", O_RDONLY) = 4
> fstat(4, 0xEFFFF04C) = 0
> mmap(0xEF7B0000, 4096, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) =
> 0xEF7B0000
> mmap(0x00000000, 94208, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF770000
> munmap(0xEF777000, 61440) = 0
> mmap(0xEF786000, 3136, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4,
> 24576) = 0xEF786000
> close(4) = 0
> open("/usr/lib/libdl.so.1", O_RDONLY) = 4
> fstat(4, 0xEFFFF04C) = 0
> mmap(0xEF7B0000, 4096, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) =
> 0xEF7B0000
> close(4) = 0
> *****************************************************************************
> open("/usr/platform/SUNW,SPARCstation-5/lib/libc_psr.so.1", O_RDONLY) Err#2
> ENOENT
> *****************************************************************************
> close(3) = 0
> brk(0x00026C38) = 0
> brk(0x00028C38) = 0
> getuid() = 0 [0]
> stat("/opt/AEgs", 0xEFFFF150) = 0
> readlink("/opt", 0xEFFFED74, 1025) Err#22 EINVAL
> readlink("/opt/AEgs", 0xEFFFED74, 1025) Err#22 EINVAL
> access("/usr/lib/fs/nfs/mount", 0) = 0
> execve("/usr/lib/fs/nfs/mount", 0xEFFFF2FC, 0xEFFFF3BC) argc = 5
> open("/dev/zero", O_RDONLY) = 3
> mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE, 3, 0) =
> 0xEF7C0000
> open("/usr/lib/librpcsvc.so.1", O_RDONLY) = 4
> fstat(4, 0xEFFFF05C) = 0
> mmap(0x00000000, 4096, PROT_READ|PROT_EXEC, MAP_SHARED, 4, 0) = 0xEF7B0000
> mmap(0x00000000, 90112, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF790000
> munmap(0xEF796000, 61440) = 0
> mmap(0xEF7A5000, 1280, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4,
> 20480) = 0xEF7A5000
> close(4) = 0
> open("/usr/lib/libnsl.so.1", O_RDONLY) = 4
> fstat(4, 0xEFFFF05C) = 0
> mmap(0xEF7B0000, 4096, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) =
> 0xEF7B0000
> mmap(0x00000000, 524288, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF700000
> munmap(0xEF760000, 61440) = 0
> mmap(0xEF76F000, 36557, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED,
> 4, 389120) = 0xEF76F000
> mmap(0xEF778000, 31408, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED,
> 3, 0) = 0xEF778000
> close(4) = 0
> open("/usr/lib/libintl.so.1", O_RDONLY) = 4
> fstat(4, 0xEFFFF05C) = 0
> mmap(0xEF7B0000, 4096, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) =
> 0xEF7B0000
> mmap(0x00000000, 81920, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF6E0000
> munmap(0xEF6E3000, 61440) = 0
> mmap(0xEF6F2000, 2416, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4,
> 8192) = 0xEF6F2000
> mmap(0xEF6F3000, 920, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3,
> 0) = 0xEF6F3000
> close(4) = 0
> open("/usr/lib/libc.so.1", O_RDONLY) = 4
> fstat(4, 0xEFFFF05C) = 0
> mmap(0xEF7B0000, 4096, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) =
> 0xEF7B0000
> mmap(0x00000000, 618496, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF640000
> munmap(0xEF6BE000, 61440) = 0
> mmap(0xEF6CD000, 28768, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED,
> 4, 512000) = 0xEF6CD000
> mmap(0xEF6D5000, 4792, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3,
> 0) = 0xEF6D5000
> close(4) = 0
> open("/usr/lib/libdl.so.1", O_RDONLY) = 4
> fstat(4, 0xEFFFF05C) = 0
> mmap(0xEF7B0000, 4096, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) =
> 0xEF7B0000
> close(4) = 0
> open("/usr/lib/libmp.so.1", O_RDONLY) = 4
> fstat(4, 0xEFFFF05C) = 0
> mmap(0x00000000, 4096, PROT_READ|PROT_EXEC, MAP_SHARED, 4, 0) = 0xEF630000
> mmap(0x00000000, 77824, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF610000
> munmap(0xEF613000, 61440) = 0
> mmap(0xEF622000, 3318, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4,
> 8192) = 0xEF622000
> close(4) = 0
> open("/usr/lib/libw.so.1", O_RDONLY) = 4
> fstat(4, 0xEFFFF05C) = 0
> mmap(0xEF630000, 4096, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_FIXED, 4, 0) =
> 0xEF630000
> mmap(0x00000000, 94208, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0xEF5F0000
> munmap(0xEF5F7000, 61440) = 0
> mmap(0xEF606000, 3136, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4,
> 24576) = 0xEF606000
> close(4) = 0
> *****************************************************************************
> open("/usr/platform/SUNW,SPARCstation-5/lib/libc_psr.so.1", O_RDONLY) Err#2
> ENOENT
> *****************************************************************************
> close(3) = 0
> munmap(0xEF630000, 4096) = 0
> getuid() = 0 [0]
> brk(0x00025778) = 0
> brk(0x00027778) = 0
> open("/etc/netconfig", O_RDONLY) = 3
> fstat(3, 0xEFFFE998) = 0
> brk(0x00027778) = 0
> brk(0x00029778) = 0
> ioctl(3, TCGETA, 0xEFFFE924) Err#25 ENOTTY
> read(3, " #\n # T h e " N e t".., 8192) = 1064
> read(3, 0x00026194, 8192) = 0
> lseek(3, 0, SEEK_CUR) = 1064
> lseek(3, 0, SEEK_SET) = 0
> read(3, " #\n # T h e " N e t".., 8192) = 1064
> read(3, 0x00026194, 8192) = 0
> lseek(3, 0, SEEK_CUR) = 1064
> close(3) = 0
> open("/etc/.name_service_door", O_RDONLY) = 3
> fcntl(3, F_SETFD, 0x00000001) = 0
> door_info(3, 0xEF6D4F18) = 0
> door_call(3, 0xEFFFCCB0, 0x00002000, 0x00000062, 0x00000000) = 0
> open("/dev/udp", O_RDWR) = 4
> ioctl(4, I_FIND, "timod") = 0
> sigfillset(0xEF6D571C) = 0
> sigprocmask(SIG_BLOCK, 0xEFFFECD0, 0xEFFFED80) = 0
> ioctl(4, I_PUSH, "timod") = 0
> sigprocmask(SIG_SETMASK, 0xEFFFED70, 0xEFFFED60) = 0
> ioctl(4, I_STR, 0xEFFFEB78) = 0
> sigprocmask(SIG_SETMASK, 0xEFFFED60, 0xEFFFED40) = 0
> ioctl(4, I_FLUSH, FLUSHRW) = 0
> sigprocmask(SIG_SETMASK, 0xEFFFED80, 0x00000000) = 0
> sigprocmask(SIG_BLOCK, 0xEFFFECC0, 0xEFFFED60) = 0
> ioctl(4, I_STR, 0xEFFFECC0) = 0
> sigprocmask(SIG_SETMASK, 0xEFFFED60, 0x00000000) = 0
> getrlimit(RLIMIT_NOFILE, 0xEFFFECA8) = 0
> brk(0x00029778) = 0
> brk(0x0002D778) = 0
> sigprocmask(SIG_BLOCK, 0xEFFFEBB0, 0xEFFFEC80) = 0
> ioctl(4, I_STR, 0xEFFFEBB0) = 0
> sigprocmask(SIG_SETMASK, 0xEFFFEC80, 0x00000000) = 0
> brk(0x0002D778) = 0
> brk(0x0004D778) = 0
> getpid() = 505 [504]
> sigprocmask(SIG_BLOCK, 0xEFFFEB20, 0xEFFFEC80) = 0
> ioctl(4, I_STR, 0xEFFFEC44) = 0
> sigprocmask(SIG_SETMASK, 0xEFFFEC80, 0x00000000) = 0
> putmsg(4, 0xEFFFECDC, 0xEFFFEE44, 0) = 0
> poll(0x0002CDBC, 1, 15000) = 1
> getmsg(4, 0xEFFFECF8, 0x00027B40, 0xEFFFED44) = 0
> putmsg(4, 0xEFFFEE8C, 0xEFFFEFF4, 0) = 0
> poll(0x0002CDBC, 1, 10000) = 1
> getmsg(4, 0xEFFFEEA8, 0x00027B40, 0xEFFFEEF4) = 0
> getuid() = 0 [0]
> sigprocmask(SIG_BLOCK, 0xEFFFEE08, 0xEFFFEEB8) = 0
> ioctl(4, I_NREAD, 0xEFFFEE14) = 0
> ioctl(4, I_STR, 0xEFFFEE08) = 0
> sigprocmask(SIG_SETMASK, 0xEFFFEEB8, 0x00000000) = 0
> ioctl(4, I_FLUSH, FLUSHRW) = 0
> sigprocmask(SIG_BLOCK, 0xEFFFED58, 0xEFFFEEB8) = 0
> ioctl(4, I_STR, 0xEFFFEE7C) = 0
> sigprocmask(SIG_SETMASK, 0xEFFFEEB8, 0x00000000) = 0
> sigprocmask(SIG_BLOCK, 0xEFFFED58, 0xEFFFEEB8) = 0
> ioctl(4, I_STR, 0xEFFFEE7C) = 0
> sigprocmask(SIG_SETMASK, 0xEFFFEEB8, 0x00000000) = 0
> getpid() = 505 [504]
> sigprocmask(SIG_BLOCK, 0xEFFFEE18, 0xEFFFEEB8) = 0
> ioctl(4, I_STR, 0xEFFFEE18) = 0
> sigprocmask(SIG_SETMASK, 0xEFFFEEB8, 0x00000000) = 0
> uname(0xEFFFEA10) = 1
> getuid() = 0 [0]
> getgid() = 1 [1]
> getgroups(16, 0xEFFFEF78) = 12
> putmsg(4, 0xEFFFEF0C, 0xEFFFF074, 0) = 0
> poll(0x0002CDBC, 1, 15000) = 1
> getmsg(4, 0xEFFFEF28, 0x00027B40, 0xEFFFEF74) = 0
> brk(0x0004D778) = 0
> brk(0x0004F778) = 0
> *****************************************************************************
> open("/usr/dt/lib/nls/msg/C/SUNW_OST_OSLIB.cat", O_RDONLY) Err#2 ENOENT
> open("/usr/lib/locale/C/LC_MESSAGES/SUNW_OST_OSLIB.mo", O_RDONLY) Err#2 ENOENT
> Permission deniedwrite(2, " P e r m i s s i o n d".., 17) = 17
> *****************************************************************************
>
> write(2, "\n", 1) = 1
> sigprocmask(SIG_SETMASK, 0xEFFFF028, 0xEFFFF018) = 0
> close(4) = 0
> sigprocmask(SIG_SETMASK, 0xEFFFF018, 0xEFFFEFF8) = 0
> lseek(0, 0, SEEK_CUR) = 107132
> _exit(33)
>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> David S. Foster Univ. of California, San Diego
> Programmer/Analyst Brain Image Analysis Laboratory
> foster@bial1.ucsd.edu Department of Psychiatry
> (619) 622-5892 8950 Via La Jolla Drive, Suite 2240
> La Jolla, CA 92037
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
David S. Foster Univ. of California, San Diego
Programmer/Analyst Brain Image Analysis Laboratory
foster@bial1.ucsd.edu Department of Psychiatry
(619) 622-5892 8950 Via La Jolla Drive, Suite 2240
La Jolla, CA 92037
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Comments
Got something to say?
You must be logged in to post a comment.
