• [6.8.0 to 6.8.3] scandir d_type and stat errors


    bashNinja
    • Closed

    Background Info

    I've been troubleshooting an application running in a docker container. I pulled it out of the docker container and ran on baremetal and am having the same issues. You can see my and others' troubleshooting here:

    https://github.com/aldostools/webMAN-MOD/issues/333
     

    We started having issues with UnRaid 6.8.0 and higher. I limited it down to a small `.cpp` files to highlight the issue.

    Small Files:
    https://gist.github.com/miketweaver/92c61293f16ef3016f9a57472fff1ff3
     

    Here is my filesystem:

    root@Tower:~# tree -L 3 /mnt/user/games/
    /mnt/user/games/
    └── GAMES/
        └── BLUS31606/
            ├── PS3_DISC.SFB
            ├── PS3_GAME/
            └── PS3_UPDATE/
    
    4 directories, 1 file ( / added by me to highlight what are dirs )
    root@Tower:~# tree -L 3 /mnt/disk*/games/
    /mnt/disk2/games/
    └── GAMES/
        └── BLUS31606/
            ├── PS3_DISC.SFB
            ├── PS3_GAME/
            └── PS3_UPDATE/
    /mnt/disk3/games/
    └── GAMES/
        └── BLUS31606
            └── PS3_GAME/
    
    7 directories, 1 file  ( / added by me to highlight what are dirs )

    As you can see, I have files spread apart across 2 disks.

    1. Scandir Issue
    When I compile `scandir.cpp` and run it on 6.7.2 (good). I get this output:

    PS3_UPDATE, d_type: DT_DIR
    PS3_GAME, d_type: DT_DIR
    PS3_DISC.SFB, d_type: DT_REG
    .., d_type: DT_DIR
    ., d_type: DT_DIR

    When I take the same file and run on 6.8.0 and higher (bad). I get this output:

    PS3_UPDATE, d_type: DT_UNKNOWN
    PS3_GAME, d_type: DT_UNKNOWN
    PS3_DISC.SFB, d_type: DT_UNKNOWN
    .., d_type: DT_UNKNOWN
    ., d_type: DT_UNKNOWN

    It appears that the changes to SHFS in 6.8.0 have caused it so that `d_type` isn't supported with `shfs`


    2. Stat Issue
    When I compile `stat.cpp` and run it on 6.7.2 (good). I get this output (every time):

    root@Tower:~# ./stat.o
    PS3_UPDATE,
    PS3_GAME,
    PS3_DISC.SFB,
    .., unknown is a Directory!
    ., unknown is a Directory!


    When I take the same file and run on 6.8.0 and higher (bad). I get this output, it varies every time I run it:

    root@Tower:~# ./stat.o
    PS3_UPDATE, unknown is a File!
    PS3_GAME, unknown is a File!
    PS3_DISC.SFB, unknown is a File!
    .., unknown is a Directory!
    ., unknown is a Directory!
    root@Tower:~# ./stat.o
    PS3_UPDATE,
    PS3_GAME,
    PS3_DISC.SFB,
    .., unknown is a Directory!
    ., unknown is a Directory!
    root@Tower:~# ./stat.o
    PS3_UPDATE,
    PS3_GAME,
    PS3_DISC.SFB,
    .., unknown is a Directory!
    ., unknown is a Directory!
    root@Tower:~# ./stat.o
    PS3_UPDATE, unknown is a FIFO!
    PS3_GAME, unknown is a FIFO!
    PS3_DISC.SFB, unknown is a FIFO!
    .., unknown is a Directory!
    ., unknown is a Directory!
    root@Tower:~# ./stat.o
    PS3_UPDATE, unknown is a CHR!
    PS3_GAME, unknown is a CHR!
    PS3_DISC.SFB, unknown is a CHR!
    .., unknown is a Directory!
    ., unknown is a Directory!

    Please visit Tools/Diagnostics and attach the diagnostics.zip file to your post.

     

    Plugins: I have many plugins, but I tested this in a virtualized unraid with 0 plugins and it has the same issue.

     

    Bare Metal Only: Yes. This is bare metal

    tower-diagnostics-20200524-1139.zip




    User Feedback

    Recommended Comments

    I'm curious if there is any update on this bug report? Has it been received? Do you plan on fixing it?

    If not, I need to pull specific applications off the Community Applications and I need to move from UnRaid to another platform.

     

    I've upgraded the report to Urgent since it's a "show stopper" and makes the array unusable for certain applications.

    Edited by bashNinja
    Link to comment

    For scandir() it's possible and valid for d_type to return DT_UNKNOWN.  See man page for readdir().  In looking at shfs nothing changed in that area and it looks like it should work.  I traced the code starting in shfs, through FUSE, then started looking at kernel.  Somewhere in the path that d_type field is set to 0 - not sure where this is happening yet - could also be in glibc.  All those subsystems changed between 6.7 and 6.8, and I see same behavior in 6.9.

     

    However 'stat' definitely works correctly but your 'stat' example contains a programming error.

    On line 12 the directory is "hard coded":

    n = scandir("/mnt/user/games/GAMES/BLUS31606", &namelist, NULL, alphasort);

    On line 19 we have:

    stat(namelist[n]->d_name,&filestat);

    The first argument passed to stat is simply the file name, not absolute path, and further, the return value of stat() call is not being checked for error.  If it was, you would see that it's returning ENOENT (and stat data is undefined) unless your current directory is set to same directory as on line 12.

     

    In summary, not sure why d_type is not being passed correctly in scandir dirent's.  But in any case proper code should check for DT_UNKNOWN and explicitly call stat() on absolute pathname of file if so.

    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.