From reiserfs to xfs : missing files!

[642]

OK. I've changed nine 4TB disk from reiserfs to xfs. It was done on a 6.3.5 unraid system, completely offline - not connected whatsoever. I've made the copy with midnight commander, locally, with a keyboard and a screen. Everything seems fine, at first.

 

After a while I discover that some files are missing.

In fact they doesn't appear on windows. For example I can see only 9 files on a directoty with 65 files (from AE01 to AE65, easy). WTF?

In fact, the problem is not as catastrophic as it seems at first, the files are still on the hdd but not visible.

I've access to tower with putty, and using mc I can see all the files, including the missing one.

It seems that there is an issue with the name, I've used some special characters like "¹" (upperscore 1) on some files, and those files doesn't appear anymore on windows.

Inside midnight commander, the files appear with the complete name, length, date, etc. However, the character "¹" (upperscore 1) was replaced by a character "empty square" in each files that doesn't appear in windows. It was probably because of the change from reiserfs to xfs.

With mc, the cure is easy, just use <shift><F6> to rename the file, going to the "empty square" character, replace it by another one and voila, the file directly appear back in windows.

If I browse through the same directory with the web page of unraid server pro, index of the disk , the files are there but without any name at all, the line is empty starting with a correct icon depending of the extension (even if there is no name or extension at all), no name and no extension, the correct size and date.

 

I don't know if ther is no other files missing with another special character in the name (like "¢", or "©" etc)

I don't know how to look inside 5203 directories for the 361902 files of 31,8 TB where are the "bad named" files.

[trurl]

Have you tried looking at them from the command line? Just wondering how they are displayed that way.

 

I have an idea that you may be looking at some advanced bash scripting with a find command, but I'm afraid I can't help there.

[bonienl]

You can start with a simple find command per disk, e.g.

find /mnt/disk1 -type d
find /mnt/disk1 -type f

The first command shows all folders on disk1,  while the second shows all files. This might give already some clues when repeated for all disks.

 

It looks like copying wasn't done using UTF-8 encoding and file (folder) names ended up with 'raw' character codes.

[trurl]

And of course, it's generally a bad idea to use these sorts of special characters anyway, especially when you will have different systems working with them.

[pwm]

Either the copy program has goofed, or the file systems are announced with the wrong character encoding.

 

The normal result of invalid UTF is that web pages do not show anything - it is considered best practices to not show invalid UTF codes because they are often used to trick users into seeing falsified information.

[642]

4 hours ago, trurl said:

Have you tried looking at them from the command line? Just wondering how they are displayed that way.

 

In fact, the files are diplayed as :

AE11\251.avi

AE13\251.avi

....

in fact the character \251 is indeed the character used for entering the "¹" (upperscore 1). 

 

If I want to correct my problem, I have to find all the files containing a character \251 and swap it for something else, like a space \032 or whatever, and this for all the directory of the disk. 

Is it something easy?

 

Anyway, the problem is strange : if I currently name a file with a "¹" into the name in windows, the file look correctly either inunraid (mc)  or in windows. If I make a copy of the file with mc, the copy seems perfecly fine. It seems that is on the step from reiserfs to xfs that there is a problem.

 

 

[642]

Hello, still stuck with the problem, those files are always missing from the directories, but in fact still on the server physically.

I want to have a solution, and a way to find all the files with the character \251 on it's name to correct those files, not by looking on all the directories one by one (more than 5000).

I may use the "script" on one disk after another, or anyway. 

I'm using PuTTY 0.60 on a windows system.

 

On 3/21/2018 at 8:25 PM, trurl said:

And of course, it's generally a bad idea to use these sorts of special characters anyway, especially when you will have different systems working with them.

Not see anything about that on the recommendation when upgrading. On my pov, there is only one system using it, and it's windows.

 

 

Edited January 1, 2019 by 642

[trurl]

On 3/21/2018 at 3:25 PM, trurl said:

And of course, it's generally a bad idea to use these sorts of special characters anyway, especially when you will have different systems working with them.

6 hours ago, 642 said:

Not see anything about that on the recommendation when upgrading. On my pov, there is only one system using it, and it's windows. 

Consider it a recommendation then. Many people who have worked with multiple systems have had experiences like this. I said "generally" because these sorts of problems occur in other situations than just changing filesystems. It is likely to bite you again in another situation if you continue to do it.

 

And of course you have more than one system here. You have at least 2 computers involved, 2 operating systems and possibly 3 filesystems.

 

This google search seems to turn up some ideas:

 

"linux find filenames with special characters"

 

Here is another suggested by that one that also has some results:

 

"remove special characters from filename linux"

 

[642]

The problem is that I haven't a 'linux' system, but only a PuTTY access. And I didn't find a way to write (write with the keyboard, so simple as that) the special character \251 that appear one time as a  "¹" (upperscore 1), somtime as an empty square, sometime as a tilde something. 

And this is only one character I've used, there are certainly others. 

How may I have a "clean" interface, on which I can write what I'm searching clearly.

 

"linux find filenames with special characters"

"remove special characters from filename linux"

 

Already tried some tricks from those results 8 months ago, but without any success. I'm not a linux's user, either. It's not my goal.

[trurl]

Unraid is linux, and when you putty to your Unraid server you are at a linux command line.

 

Can you give us more details about what you tried?