not sure what to do next...any help would be appreciated

May 3, 201016 yr

Hello (newb here...),

I noticed that I had a few drive errors recently (notably, write errors with my parity drive). It had a red ball next to it. So before I rebooted my server, I tried to copy my syslog, but was unable to write to my flash drive (Input/output error when trying to copy it). I also couldn't access it from the share. I tried saving it to another directory on the server but it wasn't there after I rebooted (I think I tried the home directory).

After securing the SATA cables, I started up the server again, and parity was starting to get rebuilt. All of the drives appeared as mounted. After I refreshed, I noticed that there was an issue with one of my data disks. It's showing up as 'Unformatted' now.

I'm still having issues with getting my syslog.

Have I lost all the data on this drive? I'm pretty sure my Parity isn't valid anymore. I know I'm not giving much to work with, but as I had mentioned I wasn't able to get my syslogs. Also, any ideas why I'm getting input/output errors on my flash drive?

Any help would be appreciated....thanks

May 3, 201016 yr

Step 1, with all the different errors and problems you're reporting you need to first perform a full memory check.

It is available as a choice on the boot menu when you first boot the array. Let it run for several cycles or better yet, overnight.

Many BIOS attempt to set the memory voltage, timing, and clock speed for you. Some get it right, many do not. Some motherboards must have specific memory installed and "premium" brands demand settings they cannot accommodate.

Then, to get to your syslog it can be as simple as entering

//tower/log/syslog

in your browser.

May 3, 201016 yr

Author

Ran memory test overnight - no errors reported.

Cannot access my syslog from my browser either. I just get a blank screen. I just checked if I could access my flash drive from the mounted share, and I could. I telnetted in, copied the syslog to /boot, but right after I did that, the files on the 'flash' share disappear. After checking the flash drive manually, the file wasn't written. Do I have a bad flash drive?

This has happened once before already with a disk becoming unformatted. I have also seen my parity drive get write errors to it before. Usually reseating the SATA cables helped. Maybe my backplane is somehow bad or faulty (ICY DOCK)?

BTW - thanks Joe, any suggestions on what I should do next? Is there any way I can remove the 'unformatted' drive and read it using a separate Linux installation or box? (took a screenshot of my disk statuses, not that it would help, but at least you can see what I'm seeing.

I know this all sounds strange, so please be patient with me as I troubleshoot! I'm very grateful for any help / insight.

Thanks

May 3, 201016 yr

Ran memory test overnight - no errors reported.

A very good first step.

Cannot access my syslog from my browser either. I just get a blank screen. I just checked if I could access my flash drive from the mounted share, and I could. I telnetted in, copied the syslog to /boot, but right after I did that, the files on the 'flash' share disappear. After checking the flash drive manually, the file wasn't written. Do I have a bad flash drive?

Working blind is not easy. Temporally move the flash drive to a PC. Run "checkdisk" or "scandisk" on it. Let's make sure it does not have file0system corruption prventing it from being written.

This has happened once before already with a disk becoming unformatted.

Unfortunatly, unRAID will show "unformattd" if the disk is unable to be mounted. It does not indicate it is actually "unformatted" and likely it is fine. Now, it might have file-system corruption that prevents it from being mounted, but that is why I asked you check the memory first. You seem to have many un-related issues. (flash drive & disk drive)

I have also seen my parity drive get write errors to it before. Usually reseating the SATA cables helped. Maybe my backplane is somehow bad or faulty (ICY DOCK)?

Anything is possible. Even the power supply.

BTW - thanks Joe, any suggestions on what I should do next? Is there any way I can remove the 'unformatted' drive and read it using a separate Linux installation or box? (took a screenshot of my disk statuses, not that it would help, but at least you can see what I'm seeing.

Yes, you can mount the drive as a reiserfs in any linux box. But first, you should be able to get the existing server running.

Just so I know for sure, what version of unRAID are you running?

I know this all sounds strange, so please be patient with me as I troubleshoot! I'm very grateful for any help / insight.

Thanks

Nothing sounds strange. It is all just clues....

Depnding on how you telnet in, you might be able to telnet in, set the telnet command to buffer 3000 lines of output, then type

tail -3000 /var/log/syslog

I don't know if the window's telnet can change the scroll-back buffer size, but if you use "putty" you can set the scroll-back buffer and capture the output that way.

You can also type these few commands to help know what is happening

mount

mkdir /mnt/disk4

mount -t reiserfs /dev/md4 /mnt/disk4

The first will see what is currently mounted.

The next two will attempt to mount disk4.

May 3, 201016 yr

Author

Working blind is not easy. Temporally move the flash drive to a PC. Run "checkdisk" or "scandisk" on it. Let's make sure it does not have file0system corruption prventing it from being written.

ran chkdsk - no errors reported.

Anything is possible. Even the power supply.

I mentioned my backplane because it always seems to be that 1 slot that gives me problems, and now the one next to it. They are also both plugged into adjacent SATA connectors on the mobo.

Yes, you can mount the drive as a reiserfs in any linux box. But first, you should be able to get the existing server running.
Just so I know for sure, what version of unRAID are you running?

4.4.2

Nothing sounds strange. It is all just clues....

Depnding on how you telnet in, you might be able to telnet in, set the telnet command to buffer 3000 lines of output, then type

tail -3000 /var/log/syslog

I don't know if the window's telnet can change the scroll-back buffer size, but if you use "putty" you can set the scroll-back buffer and capture the output that way.

You can also type these few commands to help know what is happening

mount

mkdir /mnt/disk4

mount -t reiserfs /dev/md4 /mnt/disk4

The first will see what is currently mounted.

The next two will attempt to mount disk4.

okay - did the tail trick and am attaching the output. I'm also pasting in the output of the mount commands:

root@DIRECTO_MEDIA:~# mount

fusectl on /sys/fs/fuse/connections type fusectl (rw)

usbfs on /proc/bus/usb type usbfs (rw)

/dev/sde1 on /boot type vfat (rw,umask=077,shortname=mixed)

/dev/md9 on /mnt/disk9 type reiserfs (rw,noatime,nodiratime)

/dev/md5 on /mnt/disk5 type reiserfs (rw,noatime,nodiratime)

/dev/md8 on /mnt/disk8 type reiserfs (rw,noatime,nodiratime)

/dev/md1 on /mnt/disk1 type reiserfs (rw,noatime,nodiratime)

/dev/md2 on /mnt/disk2 type reiserfs (rw,noatime,nodiratime)

/dev/md6 on /mnt/disk6 type reiserfs (rw,noatime,nodiratime)

/dev/md7 on /mnt/disk7 type reiserfs (rw,noatime,nodiratime)

/dev/md10 on /mnt/disk10 type reiserfs (rw,noatime,nodiratime)

/dev/md12 on /mnt/disk12 type reiserfs (rw,noatime,nodiratime)

/dev/md11 on /mnt/disk11 type reiserfs (rw,noatime,nodiratime)

/dev/md3 on /mnt/disk3 type reiserfs (rw,noatime,nodiratime)

/dev/md13 on /mnt/disk13 type reiserfs (rw,noatime,nodiratime)

/dev/md14 on /mnt/disk14 type reiserfs (rw,noatime,nodiratime)

shfs on /mnt/user type fuse.shfs (rw,nosuid,nodev)

root@DIRECTO_MEDIA:~# mount -t reiserfs /dev/md4 /mnt/disk4

mount: wrong fs type, bad option, bad superblock on /dev/md4,

missing codepage or helper program, or other error

In some cases useful info is found in syslog - try

dmesg | tail or so

Let me know if any of this helps the diagnosis.

I'm still having issues with the flaky flash drive in terms of not being able to write to it, and the files disappearing whenever I try to write to the mount.

syslog-20100503-1009.txt

May 3, 201016 yr

OK, the ability to get to the syslog through the browser did not get added to unRAID until the 4.5 series. That is why it did work.

The errors in syslog for disk4 are here... Odds are the file-system is corrupted.

May 3 09:05:05 DIRECTO_MEDIA kernel: ReiserFS: md4: warning: sh-2006: read_super_block: bread failed (dev md4, block 2, size 4096)

May 3 09:05:05 DIRECTO_MEDIA kernel: ReiserFS: md4: warning: sh-2006: read_super_block: bread failed (dev md4, block 16, size 4096)

May 3 09:05:05 DIRECTO_MEDIA kernel: ReiserFS: md4: warning: sh-2021: reiserfs_fill_super: can not find reiserfs on md4

I see no signs that the hard disk itself is having any problems, but you can check it by typing

smartctl -a -d ata /dev/sdk

If the disk is OK, and the file-system on it needs repair, to fix it, follow the instructions here in the wiki. You don't need to un-mount it, since it is not currently mounted, so all you need to do is

reiserfsck /dev/md4

Answer "Yes" when prompted. (Capital "Y" lower case "es") Then follow its advice. It will likely ask for you to run it again with a --fix-fixable option. If it is unable to find the superblock then come back to the forum for advice, since repairing it requires you to respond to its prompts and the default choices are NOT the correct ones for unRAID.

Joe L.

May 3, 201016 yr

Author

I'm attaching the output of the SmartCTL, as well as the Reiserfsck...

Not exactly sure how to read it, but seems like the SMART info shows no errors, but the reiserfsck aborted because it could not read a block.

Again, you're continued help is much appreciated...

smartctl_reiserfsck_output.txt

May 3, 201016 yr

Actually, the "short" test seemed to have been aborted.

Could you try requesting a smart "short" test., then wait a few minutes and get another smart status report.

I'm interesting in this section of the report:

Warning: ATA Specification requires self-test log structure revision number = 1

Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error

# 1 Extended offline Aborted by host 90% 9631 -

# 2 Short offline Aborted by host 00% 7599 -

If you cannot perform a "short" test, the disk might just be un-readable.

You can also try something like this to test if the disk is readable:

dd if=/dev/sdk count=192 | od -x -A d

and this

hdparm -I /dev/sdk

May 3, 201016 yr

Author

Still cryptic to me...but here's the output...

smarttest_take_2.txt

May 3, 201016 yr

Still cryptic to me...but here's the output...

It's ok, some of it is cryptic to me too.

The hdparrm -I output looks normal.

The smart "short" test does not seem to want to be run. Perhaps it is not supported by that drive.

The "dd" output looks pretty normal.

Normally we'd run the file-system repair on /dev/mdX to allow the repairs to automatically fix parity at the same time.

Your disk does not seem to want to be read when accessed through the "md" device.

Let's see if it can be accessed through the native "sd" device

Try;

reiserfsck /dev/sdk1

(Note, that is a "numeral one" at the end of the line above... we want to check the first partition, and its name is /dev/sdk1 )

Let's see if it still complains about not being able to read the disk.

May 3, 201016 yr

Author

Here's the reiserfsck output....

Hoping it helps...

reiserfsck_output.txt

May 3, 201016 yr

Here's the reiserfsck output....

Hoping it helps...

That is encouraging...

let's try rebooting... see if the disk will be recognized.

May 3, 201016 yr

Author

no such luck! still showing up as unformatted...

May 3, 201016 yr

no such luck! still showing up as unformatted...

Again, unformatted simply means not mounted. Do not press the "Format" button.

Let me look again at the syslog. Let me make sure we are diagnosing the correct disk.

On the "devices" page, is the disk that is failing showing as "(sdk)"

May 3, 201016 yr

Author

I haven't touched anything

disk4 device: pci-0000:00:1f.2-scsi-0:0:0:0 (sdk) ata-SAMSUNG_HD753LJ_S13UJ1FQ200089

I'm also attaching a screenshot of the devices page (just the top few disks which include the problem ones)...

May 4, 201016 yr

Author

Hey Joe - should I take a more recent snapshot of my syslog?

May 4, 201016 yr

Hey Joe - should I take a more recent snapshot of my syslog?

yes... please.

May 4, 201016 yr

Author

Here it is!

Saw a lot of weird duplicate object entries...not sure what that means...

syslog_20100504_1756.zip

May 6, 201016 yr

Author

did that last syslog help any?

May 6, 201016 yr

Author

I'm guessing the syslog wasn't of any help? Should I try to get the data through linux?

May 6, 201016 yr

I'm guessing the syslog wasn't of any help? Should I try to get the data through linux?

Sorry. did not get back to this thread until now. I'll look at the syslog now.

Joe L.

May 6, 201016 yr

Author

Please don't apologize! You're helping ME out!

Let me know if you turn anything up

May 8, 201016 yr

OK,

reading through the syslog, all look pretty normal until this:

May 3 14:08:41 DIRECTO_MEDIA emhttp: shcmd (15): mount -t reiserfs -o noatime,nodiratime /dev/md4 /mnt/disk4 >/dev/null 2>&1

May 3 14:08:41 DIRECTO_MEDIA kernel: ReiserFS: md4: warning: sh-2006: read_super_block: bread failed (dev md4, block 2, size 4096)

May 3 14:08:41 DIRECTO_MEDIA kernel: ReiserFS: md4: warning: sh-2006: read_super_block: bread failed (dev md4, block 16, size 4096)

May 3 14:08:41 DIRECTO_MEDIA kernel: ReiserFS: md4: warning: sh-2021: reiserfs_fill_super: can not find reiserfs on md4

We've proven earlier that disk4 does indeed have a file-system on its first partition... so this is a mystery why it cannot be accessed through the "md" device.

Then, we see this. The USB device is reset.

May 3 14:08:41 DIRECTO_MEDIA kernel: usb 2-2: reset high speed USB device using ehci_hcd and address 3

Followed by this... Your flash drive is now un-readable

May 3 14:08:41 DIRECTO_MEDIA kernel: usb 2-2: device descriptor read/64, error -71

May 3 14:08:41 DIRECTO_MEDIA kernel: ReiserFS: md9: Using r5 hash to sort names

May 3 14:08:41 DIRECTO_MEDIA emhttp: disk4 mount error: 32

May 3 14:08:41 DIRECTO_MEDIA kernel: can't shrink filesystem on-line

May 3 14:08:41 DIRECTO_MEDIA kernel: ReiserFS: md13: Using r5 hash to sort names

May 3 14:08:41 DIRECTO_MEDIA kernel: ReiserFS: md11: Using r5 hash to sort names

May 3 14:08:41 DIRECTO_MEDIA kernel: ReiserFS: md12: Using r5 hash to sort names

May 3 14:08:41 DIRECTO_MEDIA kernel: can't shrink filesystem on-line

May 3 14:08:41 DIRECTO_MEDIA last message repeated 9 times

May 3 14:08:41 DIRECTO_MEDIA kernel: usb 2-2: device descriptor read/64, error -71

May 3 14:08:42 DIRECTO_MEDIA kernel: usb 2-2: reset high speed USB device using ehci_hcd and address 3

May 3 14:08:42 DIRECTO_MEDIA emhttp: shcmd (14): cp /etc/exports- /etc/exports

May 3 14:08:42 DIRECTO_MEDIA emhttp: shcmd (15): mkdir -m 700 /mnt/user

May 3 14:08:42 DIRECTO_MEDIA emhttp: shcmd (16): /usr/local/sbin/shfs /mnt/user

May 3 14:08:42 DIRECTO_MEDIA kernel: usb 2-2: device descriptor read/64, error -71

May 3 14:08:42 DIRECTO_MEDIA kernel: usb 2-2: reset high speed USB device using ehci_hcd and address 3

May 3 14:08:43 DIRECTO_MEDIA kernel: usb 2-2: device not accepting address 3, error -71

May 3 14:08:43 DIRECTO_MEDIA kernel: usb 2-2: reset high speed USB device using ehci_hcd and address 3

May 3 14:08:43 DIRECTO_MEDIA kernel: usb 2-2: device not accepting address 3, error -71

May 3 14:08:43 DIRECTO_MEDIA kernel: sd 0:0:0:0: [sde] Result: hostbyte=0x07 driverbyte=0x00

May 3 14:08:43 DIRECTO_MEDIA kernel: end_request: I/O error, dev sde, sector 223183

May 3 14:08:43 DIRECTO_MEDIA kernel: sd 0:0:0:0: [sde] Result: hostbyte=0x07 driverbyte=0x00

May 3 14:08:43 DIRECTO_MEDIA kernel: end_request: I/O error, dev sde, sector 8560

May 3 14:08:43 DIRECTO_MEDIA kernel: Buffer I/O error on device sde1, logical block 8497

May 3 14:08:43 DIRECTO_MEDIA kernel: lost page write due to I/O error on sde1

May 3 14:08:43 DIRECTO_MEDIA kernel: write_file: write error 5

May 3 14:08:43 DIRECTO_MEDIA kernel: md: could not write superblock from /boot/config/super.dat

May 3 14:08:43 DIRECTO_MEDIA kernel: md: recovery thread has nothing to resync

May 3 14:08:43 DIRECTO_MEDIA kernel: usb 2-2: USB disconnect, address 3

May 3 14:08:43 DIRECTO_MEDIA kernel: sd 0:0:0:0: [sde] Result: hostbyte=0x01 driverbyte=0x00

May 3 14:08:43 DIRECTO_MEDIA kernel: end_request: I/O error, dev sde, sector 64

May 3 14:08:43 DIRECTO_MEDIA kernel: Buffer I/O error on device sde1, logical block 1

May 3 14:08:43 DIRECTO_MEDIA kernel: lost page write due to I/O error on sde1

May 3 14:08:43 DIRECTO_MEDIA kernel: Buffer I/O error on device sde1, logical block 6454

May 3 14:08:43 DIRECTO_MEDIA kernel: lost page write due to I/O error on sde1

May 3 14:08:43 DIRECTO_MEDIA kernel: Buffer I/O error on device sde1, logical block 8496

May 3 14:08:43 DIRECTO_MEDIA kernel: lost page write due to I/O error on sde1

May 3 14:08:43 DIRECTO_MEDIA kernel: sd 0:0:0:0: [sde] Result: hostbyte=0x01 driverbyte=0x00

May 3 14:08:43 DIRECTO_MEDIA kernel: end_request: I/O error, dev sde, sector 4570

May 3 14:08:43 DIRECTO_MEDIA kernel: Buffer I/O error on device sde1, logical block 4507

May 3 14:08:43 DIRECTO_MEDIA kernel: lost page write due to I/O error on sde1

May 3 14:08:43 DIRECTO_MEDIA kernel: Buffer I/O error on device sde1, logical block 8496

May 3 14:08:43 DIRECTO_MEDIA kernel: lost page write due to I/O error on sde1

Then, it is followed by this It is really trying to read the flash drive, but the usb reset really got it confused.

:

May 3 14:08:43 DIRECTO_MEDIA kernel: FAT: Directory bread(block 9448) failed

May 3 14:08:43 DIRECTO_MEDIA kernel: FAT: Directory bread(block 9449) failed

May 3 14:08:43 DIRECTO_MEDIA kernel: FAT: Directory bread(block 9450) failed

May 3 14:08:43 DIRECTO_MEDIA kernel: FAT: Directory bread(block 9451) failed

May 3 14:08:43 DIRECTO_MEDIA kernel: FAT: Directory bread(block 9452) failed

May 3 14:08:43 DIRECTO_MEDIA kernel: FAT: Directory bread(block 9453) failed

May 3 14:08:43 DIRECTO_MEDIA kernel: FAT: Directory bread(block 9454) failed

May 3 14:08:43 DIRECTO_MEDIA kernel: FAT: Directory bread(block 9455) failed

May 3 14:08:43 DIRECTO_MEDIA emhttp: get_config_idx: fopen /boot/config/shares/Television.cfg: No such file or directory - assigning defaults

May 3 14:08:43 DIRECTO_MEDIA kernel: FAT: Directory bread(block 9448) failed

May 3 14:08:43 DIRECTO_MEDIA kernel: FAT: Directory bread(block 9449) failed

May 3 14:08:43 DIRECTO_MEDIA kernel: FAT: Directory bread(block 9450) failed

May 3 14:08:43 DIRECTO_MEDIA kernel: FAT: Directory bread(block 9451) failed

May 3 14:08:43 DIRECTO_MEDIA kernel: FAT: Directory bread(block 9452) failed

May 3 14:08:43 DIRECTO_MEDIA kernel: FAT: Directory bread(block 9453) failed

May 3 14:08:43 DIRECTO_MEDIA kernel: FAT: Directory bread(block 9454) failed

May 3 14:08:43 DIRECTO_MEDIA emhttp: get_config_idx: fopen /boot/config/shares/Temporary.cfg: No such file or directory - assigning defaults

May 3 14:08:43 DIRECTO_MEDIA emhttp: shcmd (17): killall -HUP smbd

May 3 14:08:43 DIRECTO_MEDIA kernel: FAT: Directory bread(block 9455) failed

May 3 14:08:43 DIRECTO_MEDIA emhttp: shcmd (18): /etc/rc.d/rc.nfsd restart >/dev/null

May 3 14:08:43 DIRECTO_MEDIA kernel: usb 2-2: new high speed USB device using ehci_hcd and address 4

May 3 14:08:43 DIRECTO_MEDIA kernel: usb 2-2: device descriptor read/64, error -71

May 3 14:08:44 DIRECTO_MEDIA kernel: usb 2-2: device descriptor read/64, error -71

May 3 14:08:44 DIRECTO_MEDIA kernel: usb 2-2: new high speed USB device using ehci_hcd and address 5

May 3 14:08:44 DIRECTO_MEDIA kernel: usb 2-2: device descriptor read/64, error -71

May 3 14:08:44 DIRECTO_MEDIA kernel: FAT: unable to read inode block for updating (i_pos 151171)

May 3 14:08:44 DIRECTO_MEDIA kernel: usb 2-2: device descriptor read/64, error -71

May 3 14:08:44 DIRECTO_MEDIA kernel: usb 2-2: new high speed USB device using ehci_hcd and address 6

May 3 14:08:45 DIRECTO_MEDIA kernel: usb 2-2: device not accepting address 6, error -71

May 3 14:08:45 DIRECTO_MEDIA kernel: usb 2-2: new high speed USB device using ehci_hcd and address 7

May 3 14:08:45 DIRECTO_MEDIA kernel: usb 2-2: device not accepting address 7, error -71

May 3 14:08:45 DIRECTO_MEDIA kernel: hub 2-0:1.0: unable to enumerate USB device on port 2

May 3 14:08:46 DIRECTO_MEDIA kernel: usb 6-2: new full speed USB device using uhci_hcd and address 3

May 3 14:08:46 DIRECTO_MEDIA kernel: usb 6-2: device descriptor read/64, error -71

May 3 14:08:47 DIRECTO_MEDIA kernel: usb 6-2: new full speed USB device using uhci_hcd and address 4

May 3 14:08:47 DIRECTO_MEDIA kernel: usb 6-2: device descriptor read/64, error -71

May 3 14:08:47 DIRECTO_MEDIA ntpd[1466]: synchronized to 209.167.68.100, stratum 2

May 3 14:08:47 DIRECTO_MEDIA ntpd[1466]: time reset +0.324572 s

May 3 14:08:47 DIRECTO_MEDIA kernel: usb 6-2: new full speed USB device using uhci_hcd and address 5

May 3 14:08:48 DIRECTO_MEDIA kernel: usb 6-2: device not accepting address 5, error -71

May 3 14:08:48 DIRECTO_MEDIA kernel: usb 6-2: new full speed USB device using uhci_hcd and address 6

May 3 14:08:48 DIRECTO_MEDIA kernel: usb 6-2: device not accepting address 6, error -71

May 3 14:08:48 DIRECTO_MEDIA kernel: hub 6-0:1.0: unable to enumerate USB device on port 2

May 3 14:08:49 DIRECTO_MEDIA kernel: FAT: Directory bread(block 8192) failed

May 3 14:08:49 DIRECTO_MEDIA kernel: FAT: Directory bread(block 8193) failed

May 3 14:08:49 DIRECTO_MEDIA kernel: FAT: Directory bread(block 8194) failed

May 3 14:08:49 DIRECTO_MEDIA kernel: FAT: Directory bread(block 8195) failed

May 3 14:08:49 DIRECTO_MEDIA kernel: FAT: Directory bread(block 8196) failed

May 3 14:08:49 DIRECTO_MEDIA kernel: FAT: Directory bread(block 8197) failed

May 3 14:08:49 DIRECTO_MEDIA kernel: FAT: Directory bread(block 8198) failed

May 3 14:08:49 DIRECTO_MEDIA kernel: FAT: Directory bread(block 8199) failed