Having received my 'plus' key today, I thought that I would have a play with setting up a cache drive in order to evaluate the benefits of using a spare drive as cache, vs adding it to the array.

 

Now, the drive I have configured as cache is my old 'drive2' so, when I compared disk2/Movies against cache/Movies, remotely by nfs share, they looked exactly the same.

 

So, to clear the cache drive, I deleted all the files in cache/Movies, remotely over the nfs share.

 

This appeared to have the desired effect - via nfs shares, the cache drive appeared to have no files, while disk2 appeared to have all the mkv files I expected to see.

 

I then noticed that, on the web interface, the cache disk was still showing just 50MB free, while disk2 was showing almost 1TB free.

 

I then logged in via telnet and found that 'ls /mnt/disk2/Movies' listed no files, while 'ls /mnt/cache/Movies' listed all the files I thought that I had deleted.

 

Is this expected behaviour?  It certainly wasn't what I expected!  I am absolutely 100% certain that I performed the deletion on the cache share, and this was confirmed by what I was being shown in the file viewer windows.  I had fully expected that the files would be deleted from the cache drive.

 

I have now invoked 'mover' and my files are currently being moved from cache to disk2.

No, that definitely should not have happened.  I've deleted files off my cache drive many times and never seen this behavior.  I only use SMB, though, so perhaps it is an issue with NFS?

 

The most likely scenario is that you accidentally deleted the files off of disk2, though, even though you said you are sure you didn't.

 

To test this further you could temporarily set both your user and disk shares to read-only and try it again (with data that you don't care about).

Another thought on this .... the deletion happened relatively quickly, but perhaps not quite as quickly as I expected.  I'm wondering whether, as the files were deleted from disk2, the parity drive was being updated???

Anytime you delete a file from the array parity should be updated.  The cache drive is outside the array, so it has no effect on parity whatsoever.

No, that definitely should not have happened.  I've deleted files off my cache drive many times and never seen this behavior.  I only use SMB, though, so perhaps it is an issue with NFS?

 

The other factor here is that the content of the two drives was identical ... having only recently replaced the 400GB drive (now my cache) with a 1TB drive, and rebuilding the contents of disk2 from the parity.

 

The most likely scenario is that you accidentally deleted the files off of disk2, though, even though you said you are sure you didn't.

 

Admittedly, that sounds the most plausible explanation but, as I said, the file viewers on my ubuntu desktop were showing an empty cache and a populated disk2 (this was after I had spotted the anomaly in the free space).

 

To test this further you could temporarily set both your user and disk shares to read-only and try it again (with data that you don't care about).

 

Well, I guess that the first thing to try would be to add a new file to both drives (via a telnet console), and then try deleting that in the ubuntu desktop.  It's now almost 2am here ... I'll let mover complete its task, and do some experiments tomorrow.

Anytime you delete a file from the array parity should be updated.

 

True, and I hope that the parity handling is at a low-enough level that any discrepancy between array drive and cache drive writing won't negate that.  As I say, I wasn't deleting the files from the array - I was deleting from the cache!  I didn't time it, but it took a good couple of minutes - I guess that is long enough to clear 98 directory entries and update the parity to match.

 

The cache drive is outside the array, so it has no effect on parity whatsoever.

 

Exactly.

I didn't time it, but it took a good couple of minutes - I guess that is long enough to clear 98 directory entries and update the parity to match.

 

Deleting from the cache drive should be instant, so something definitely went wrong here.  Maybe capture a syslog before you reboot also.

Yes, I've just been looking at the syslog.  The first thing I notice is that REISERFS had replayed 51 transactions.

 

Then there are a lot of entries from emhttp attempting lots of umounts and rmdirs.

 

Then I notice that shfs reports lots of duplicate objects on disk2.  Now, they can't really have been duplicated on disk2 - the duplicates were on the cache drive.

 

However, just before the list of duplicate objects, I see:

Sep 3 00:11:19 Tower mountd[7713]: /mnt/disk2 and /mnt/cache have same filehandle for *, using first

 

Hmmm ... I think that might explain what was going on - the crucial question, though, is how did they come to have the same handle???

 

Ah, I've just noticed that one file I added recently is not on disk1 - it was added since I swapped the drives and must have gone on the new disk2.  It has, therefore, gone missing.

syslog-2010-09-03.txt.zip

Mover finished overnight and I've rebooted unRAID.  However, I still get the message:

Sep 3 06:19:59 Tower mountd[2751]: /mnt/disk2 and /mnt/cache have same filehandle for *, using first

in the syslog and, of course, the cache share and the disk2 share are still showing the same content!

 

How do I fix this?

 

Mover finished overnight and I've rebooted unRAID.  However, I still get the message:

Sep 3 06:19:59 Tower mountd[2751]: /mnt/disk2 and /mnt/cache have same filehandle for *, using first

in the syslog and, of course, the cache share and the disk2 share are still showing the same content!

 

How do I fix this?

 

I don't know how to fix it, but it is affiliated with how you have set up the NFS file shares.

See here for some ideas: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/436527

 

I do not use NFS, so cannot help you much more.

 

Joe L.

Ah, so the filehandle is assigned by the client, not the server?  That's not what I was expecting!

 

All my shares are set up through the File Browser gui on ubuntu, so I'm not sure what control I have over the filehandle.  However, it gives me a clue of where to continue my investigations ... thank you Joe!

Further reading does suggest that the problem is created by the server, and it seems that something must have changed in nfsd at some stage.  Reports of this problem exist for ubuntu, fedora, redhat, centos etc.  The typical report is that the problem first manifested itself after an O/S upgrade, although some appear to attribute this to 'coincidence'.

 

The advice for a workaround centres on two settings in the /etc/exports file - using 'no_root_squash' and/or setting the fsid to non-zero.

 

Now, I would be happy to try these possible solutions, but I'm not sure how to control the settings within the unRAID environment.  /etc/exports is, obviously, held in ram, so must be created at run time, presumably from the content of /boot/config/share.cfg.  Can anyone give any advice on controlling the options within the unRAID /etc/exports?

 

I'm interested to note that user shares already set a non-zero fsid - I presume because this problem has raised its head, in the context of user shares, at some time in the past.  Perhaps it would be sufficient to add an fsid setting to the cache share too?

 

As an aside, the addition of the cache drive has boosted the speed of writes to a user share - previously I was getting around 25MB/sec and the first write since adding the cache reports 48MB/sec.

AFAIK - the 'fsid' is generated by the server and represents just part of the "file handle" maintained by a client.  If the fsid is the same for two file handles, it is possible that the client can access the same server resource via two different mount points, giving the behavior you see here.

 

However, I don't see how that is happening with unRaid.

 

The way nfsd is supposed to work (again AFAIK), is that if fsid is not specified then it's 'derived' from the MAJOR/MINOR numbers of the device that the file system is mounted on.  So for disk shares (and the cache share), the fsid should be derived from the underlying physical disk major/minor which is unique across all disk partitions, and in the case of unRaid should also be persistent (that is, the same) across all reboots.

 

For user shares, it's a different story.  User shares are not mounted on any physical device, hence for user shares, unRaid must generate a fsid value for each share it exports.  Now having investigated this further, it seems like what I'm doing here is probably wrong: what it does is just assign fsid's starting at 100 and incrementing for each user share in a sorted list of users shares.  But I think it's sufficient to generated a single fsid and use for all exported user shares - I'll have to look at this further.

 

Nevertheless, the behavior you are seeing with cache and disk2 should not happen.  In your investigations, did you run across any BUG reports of this?  (This will save me some time if you can point them out.)

 

BTW, I hate NFS  

 

 

EDIT: now that I think about it, I think it's possible the nfs-generated fsid for disk/cache shares might not be persistent across server reboots.  Are you able to reproduce this behavior?

To clarify the above Edit... (and this might explain what others are seeing in ubuntu, centos, etc)...

 

This comes back to something that's been a pain-in-the-neck in linux for a while (ever since libata was introduced I think).  When a server boots, for scsi drives, the exact scsi disk identifier could change.  For example,

 

After reboot #1 we find:

diskA is assigned /dev/sda

diskB is assigned /dev/sdb

 

But after reboot #2 we find:

diskA is assigned /dev/sdb

diskB is assigned /dev/sda

 

This is because disk identifiers are assigned in order of device discovery.  This results in identifiers being somewhat arbitrary (now in practice, usually identifiers get assigned the same across reboots because disk discovery tends to be fairly regular, but especially for USB drives, discovery can be notoriously un-regular (is that a word?)).

 

What does this have to do with fsid?  Well the disk identifier directly corresponds to MAJOR/MINOR.  So let's add this to above:

 

After reboot #1:

diskA is assigned /dev/sda (internal fsid 80101)

diskB is assigned /dev/sdb (internal fsid 80201)

 

After reboot #2:

diskA is assigned /dev/sdb (internal fsid 80201)

diskB is assigned /dev/sda (internal fsid 80101)

 

Now after reboot #2, none of the client mounts have changed, hence client will access diskB thinking he's accessing diskA.

 

Does this explanation fit with what you are seeing?  The crucial thing here is there had been a unRaid reboot after having previously mounted nfs shares, without also an intervening client reboot.

 

 

Someone might ask: I don't have any scsi drives, why is this an issue?  Well SATA uses the linux SCSI driver subsystem, so to linux SATA==SCSI (and btw, USBStorage==SCSI).

 

 

Yes, this feature has persisted since the day I installed the cache drive, and as still demonstrable on 4.6-rc3.

 

This one, a telnet session to the unRAID server, shows that the cache is really empty, while disk2 has several directories:

Tower login: root
Linux 2.6.32.9-unRAID.
root@Tower:~# ls /mnt/cache
root@Tower:~# ls /mnt/disk2
101029/  Movies/  Videos/  ZA30/  ZVideo/   mediaserver/
Athlon/  Series/  Work/    ZK10/  ZVideo2/
root@Tower:~#

 

This one, a local terminal session on my desktop ubuntu machine, accessing the nfs shares on the unRAID server, makes it appear that the cache drive and disk2 have the same contents:

peter@Desktop:~$ ls /net/tower/mnt/cache
101029  mediaserver  Series  Work  ZK10    ZVideo2
Athlon  Movies       Videos  ZA30  ZVideo
peter@Desktop:~$ ls /net/tower/mnt/disk2
101029  mediaserver  Series  Work  ZK10    ZVideo2
Athlon  Movies       Videos  ZA30  ZVideo
peter@Desktop:~$ 

 

 

Here is a snippet from the unRAID logfile, showing the report of the duplicated filehandle:

Nov 19 01:44:40 Tower kernel: svc: 10.2.1.15, port=698: unknown version (4 for prog 100003, nfsd)
Nov 19 01:44:40 Tower mountd[2751]: authenticated mount request from 10.2.1.15:873 for /mnt/disk2 (/mnt/disk2)
Nov 19 01:44:46 Tower mountd[2751]: authenticated mount request from 10.2.1.15:873 for /mnt/disk2 (/mnt/disk2)
Nov 19 01:44:53 Tower mountd[2751]: /mnt/disk2 and /mnt/cache have same filehandle for *, using first
Nov 19 01:44:53 Tower kernel: svc: 10.2.1.15, port=885: unknown version (4 for prog 100003, nfsd)
Nov 19 01:44:53 Tower mountd[2751]: authenticated mount request from 10.2.1.15:864 for /mnt/disk1 (/mnt/disk1)
Nov 19 01:44:53 Tower mountd[2751]: authenticated mount request from 10.2.1.15:920 for /mnt/cache (/mnt/cache)
Nov 19 01:44:53 Tower kernel: svc: 10.2.1.15, port=995: unknown version (4 for prog 100003, nfsd)
Nov 19 01:45:11 Tower kernel: svc: 10.2.1.15, port=924: unknown version (4 for prog 100003, nfsd)

 

It might be worth pointing out that I originally experienced the problem while running ubuntu 10.04 on my desktop machine.  I have since experienced a severe failure of the Samsung disk drive on that machine and have done a fresh install of 10.10.  The problem still appears to be identical.  Furthermore, the same problem can be demonstrated on an entirely different client, running Xandros.

Does this explanation fit with what you are seeing?  The crucial thing here is there had been a unRaid reboot after having previously mounted nfs shares, without also an intervening client reboot.

 

No, this does not correspond at all with what I'm experiencing.  First of all, the most frequent reason for reboots is the frequent (average of one a day) power cuts which occur in this part of the world.  Both the unRAID server, and my ubuntu desktop machine are running from the same UPS - hence, they are almost always booted up at the same time.

 

Secondly (I will have to go back and produce the hard evidence, when I've remembered how), what I am seeing is that physical drive shares are always being allocated an fsid of zero, while the user shares are allocated a unique, non-zero, fsid.

Secondly (I will have to go back and produce the hard evidence, when I've remembered how), what I am seeing is that physical drive shares are always being allocated an fsid of zero, while the user shares are allocated a unique, non-zero, fsid.

 

Where do you see that fsid is set to 0 (that would certainly explain it)?  From telnet to server, type this command:

 

cat /etc/exports

 

should not be any "fsid=0" in there.

 

I notice ubuntu is trying to use NFSv4 which is currently not enabled in the unRaid kernel (why? because no one requested it and I hate NFS).  Is it possible to tell ubuntu to use NFSv3?

Okay, exports doesn't actually show zero fsids, but on all the physical disks, it's just not specified (I guess that means it will be left as zero?).  I know that, during my earlier investigations, I did see an explicit fsid=0 reported somewhere.

 

root@Tower:~# cat /etc/exports
# See exports(5) for a description.
# This file contains a list of all directories exported to other computers.
# It is used by rpc.nfsd and rpc.mountd.

"/mnt/disk1" -async,no_subtree_check,anongid=0,anonuid=0,all_squash *(rw,no_root_squash)
"/mnt/disk2" -async,no_subtree_check,anongid=0,anonuid=0,all_squash *(rw,no_root_squash)
"/mnt/cache" -async,no_subtree_check,anongid=0,anonuid=0,all_squash *(rw,no_root_squash)
"/mnt/user/Movies" -async,no_subtree_check,anongid=0,anonuid=0,all_squash,fsid=102 *(rw)
"/mnt/user/Series" -async,no_subtree_check,anongid=0,anonuid=0,all_squash,fsid=104 *(rw)
"/mnt/user/Videos" -async,no_subtree_check,anongid=0,anonuid=0,all_squash,fsid=109 *(rw)
root@Tower:~# 

A few comments/thoughts on this issue:

 

1) I'm still trying to recall what command it was which showed the fsids for each share (including the zeroes I remember seeing for the physical drives).

 

2) I'm not sure that there is any control over nfs3/4 at the client end ... I believe that it simply negotiates with the server.  However, if anyone knows better, I'd be very happy to change a setting to force nfs3 for a test.  However, since the problem is apparent on all clients (ubuntu, Xandros and my two PopcornHours - I'm not sure what distro these are based on), I don't believe that this would be the correct solution.  Incidently, the Popcorn allows the protocol to be set as either nfs, or nfs/tcp - changing this setting does not alter the symptoms.

 

3) I wonder why this problem is afflicting my cache and disk2 drives, but not disk1 ... is it anything to do with the fact that the cache drive used to be my disk2???.  If so, perhaps running a preclear on that (cache) drive would make the problem go away???

 

4) I'm awaiting delivery of another drive - I wonder whether the symptoms will alter when this drive is added - will disk3 become involved in the confusion?

 

5) I believe that the user share fsids are assigned by the server based simply on their position in the configuration list(100+n, where n is the position in the list of user shares, starting at zero).

 

6) Would it not be possible to do something similar with the physical drive shares - assigning the fsid as disk number plus 1, leaving fsid 1 for the cache drive.

A few comments/thoughts on this issue:

 

1) I'm still trying to recall what command it was which showed the fsids for each share (including the zeroes I remember seeing for the physical drives).

I've never seen such a command but I can guarantee that unRaid never sets fsid=0.

 

2) I'm not sure that there is any control over nfs3/4 at the client end ... I believe that it simply negotiates with the server.  However, if anyone knows better, I'd be very happy to change a setting to force nfs3 for a test.  However, since the problem is apparent on all clients (ubuntu, Xandros and my two PopcornHours - I'm not sure what distro these are based on), I don't believe that this would be the correct solution.  Incidently, the Popcorn allows the protocol to be set as either nfs, or nfs/tcp - changing this setting does not alter the symptoms.

Agreed.

 

3) I wonder why this problem is afflicting my cache and disk2 drives, but not disk1 ... is it anything to do with the fact that the cache drive used to be my disk2???.  If so, perhaps running a preclear on that (cache) drive would make the problem go away???

That is most certainly the cause of this problem.  'preclear' has nothing to do with this.  If you were able to clear the NFS file handles on the client side this issue should go away.  Not sure why a client re-boot does not do this.

 

4) I'm awaiting delivery of another drive - I wonder whether the symptoms will alter when this drive is added - will disk3 become involved in the confusion?

Probably not.  The issue isn't the drive itself, but the physical identifier assigned to the drive.

 

5) I believe that the user share fsids are assigned by the server based simply on their position in the configuration list(100+n, where n is the position in the list of user shares, starting at zero).

That's correct.  But as mentioned earlier I don't think this is necessary (to assign a unique fsid to each use share) because of the way user share filesystem is implemented.

 

6) Would it not be possible to do something similar with the physical drive shares - assigning the fsid as disk number plus 1, leaving fsid 1 for the cache drive.

Yes it would be possible and I am testing this change in 4.6-rc4.

A few comments/thoughts on this issue:

 

1) I'm still trying to recall what command it was which showed the fsids for each share (including the zeroes I remember seeing for the physical drives).

I've never seen such a command but I can guarantee that unRaid never sets fsid=0.

 

No, I'm sure that unRAID never, explicitly, sets fsid to zero, but can we be sure what happens when fsid is not set explicitly?

 

2) I'm not sure that there is any control over nfs3/4 at the client end ... I believe that it simply negotiates with the server.  However, if anyone knows better, I'd be very happy to change a setting to force nfs3 for a test.  However, since the problem is apparent on all clients (ubuntu, Xandros and my two PopcornHours - I'm not sure what distro these are based on), I don't believe that this would be the correct solution.  Incidently, the Popcorn allows the protocol to be set as either nfs, or nfs/tcp - changing this setting does not alter the symptoms.

Agreed.

 

Thank you!

 

3) I wonder why this problem is afflicting my cache and disk2 drives, but not disk1 ... is it anything to do with the fact that the cache drive used to be my disk2???.  If so, perhaps running a preclear on that (cache) drive would make the problem go away???

That is most certainly the cause of this problem.  'preclear' has nothing to do with this.  If you were able to clear the NFS file handles on the client side this issue should go away.  Not sure why a client re-boot does not do this.

 

Not just a client re-boot, but a full, fresh installation of the O/S!  Also, multiple upgrades of the unRAID system.  Perhaps a clear out of my unRAID config would fix it???

 

4) I'm awaiting delivery of another drive - I wonder whether the symptoms will alter when this drive is added - will disk3 become involved in the confusion?

Probably not.  The issue isn't the drive itself, but the physical identifier assigned to the drive.

 

... except that my plan is to install the new 2TB drive as parity and to re-deploy the present 1TB parity drive as drive3!

 

5) I believe that the user share fsids are assigned by the server based simply on their position in the configuration list(100+n, where n is the position in the list of user shares, starting at zero).

That's correct.  But as mentioned earlier I don't think this is necessary (to assign a unique fsid to each use share) because of the way user share filesystem is implemented.

 

No, I'm sure that, theoretically, it shouldn't be necessary ... however, it ought to be a good work-around for this fault.

 

6) Would it not be possible to do something similar with the physical drive shares - assigning the fsid as disk number plus 1, leaving fsid 1 for the cache drive.

Yes it would be possible and I am testing this change in 4.6-rc4.

 

Wow, you're a gentleman, sir!  Thank you very much ... I'm more than 99% certain that it will fix the problem!

 

Now, if only the likes of Microsoft would follow your lead in customer service/satisfaction, I might even be tempted to use Windows (and accept smb as my primary nfs)!

PeterB - I have modified the way NFS fsid's get assigned and this should fix the problem you are seeing.  Please send me an email: [email protected] so that I can give you a download link so that you can verify for me.