January 6, 201313 yr Good afternoon, after a parity check I have a bunch of errors in the syslog. I think, that the parity check just made it visible that something's not correct. It may be a cabling issue? Can anyone help? Thanks in advance.
January 7, 201313 yr Check the SATA cable connecting disk2. Try a new cable and/or SATA port. Once the ATA errors are fixed, see check file systems in my sig.
January 7, 201313 yr Author You are right - the SATA connection was the issue. It seems that the Sharkoon QuickPort 3-Bay (I have 3 of them) is always the delinquent. Anyhow, I had 505817 errors on the first Parity-Check run, and 508336 errors yesterday. I was running reiserfsck --check /dev/md2 (syslog attached) and will perform reiserfsck --fix-fixable /dev/md2 now.
January 7, 201313 yr Author One large file that I transferred recently seem to be corrupted, hence I deleted it. After another run reiserfsck --fix-fixable /dev/md2 the syslog reports only a few errors, but still...... What steps should I take now? How should I run the Parity-Check - with or without correction? Thanks for any further advise.
January 7, 201313 yr Author You can see the errors in the syslog that I had attached in my previous post. 19 errors are still left.
January 7, 201313 yr That's not a syslog. It says the repeat reiserfsck with the --rebuild-tree option. This should be the next action.
January 10, 201313 yr Author Thanks a lot dgaschk, it seems that there is a broader issue. unRaid is always starting with a parity sync. When I access the webGUI the server is rebooting - of course with a parity sync again. Issue then is that the webGUI isn' available. Fortunately I could access the log: Jan 10 11:00:09 Tower kernel: Call Trace: (Errors) Jan 10 11:00:09 Tower kernel: [<c10d6889>] leaf_paste_in_buffer+0x65/0x1bd (Errors) Jan 10 11:00:09 Tower kernel: [<c10c7187>] balance_leaf+0x7d0/0x1d7e (Errors) Jan 10 11:00:09 Tower kernel: [<c10cfe71>] ? check_right+0xf5/0x117 (Errors) Jan 10 11:00:09 Tower kernel: [<c10d1da1>] ? ip_check_balance+0x463/0xb35 (Errors) Jan 10 11:00:09 Tower kernel: [<c10a2385>] ? __find_get_block+0x132/0x13c (Errors) Jan 10 11:00:09 Tower kernel: [<c1035ea8>] ? wake_up_bit+0x57/0x5b (Errors) Jan 10 11:00:09 Tower kernel: [<c10c87c4>] do_balance+0x8f/0x137 (Errors) Jan 10 11:00:09 Tower kernel: [<c10d271e>] ? fix_nodes+0x2ab/0x441 (Errors) Jan 10 11:00:09 Tower kernel: [<c10dacc0>] reiserfs_paste_into_item+0x143/0x17e (Errors) Jan 10 11:00:09 Tower kernel: [<c10cd70f>] reiserfs_get_block+0xd64/0xfcd (Errors) Jan 10 11:00:09 Tower kernel: [<c10a2c03>] ? ll_rw_block+0x65/0x71 (Errors) Jan 10 11:00:09 Tower kernel: [<c10d9eed>] ? search_by_key+0x830/0xdb4 (Errors) Jan 10 11:00:09 Tower kernel: [<c10a30f2>] __block_write_begin+0x151/0x2f5 (Errors) Jan 10 11:00:09 Tower kernel: [<c10cba1d>] reiserfs_write_begin+0x113/0x19a (Errors) Jan 10 11:00:09 Tower kernel: [<c10cc9ab>] ? restart_transaction+0x86/0x86 (Errors) Jan 10 11:00:09 Tower kernel: [<c105dc46>] generic_perform_write+0x96/0x17d (Errors) Jan 10 11:00:09 Tower kernel: [<c105dd6c>] generic_file_buffered_write+0x3f/0x6b (Errors) Jan 10 11:00:09 Tower kernel: [<c105eff2>] __generic_file_aio_write+0x3a5/0x3e6 (Errors) Jan 10 11:00:09 Tower kernel: [<c103e460>] ? try_to_wake_up+0x1b0/0x1b0 (Errors) Jan 10 11:00:09 Tower kernel: [<c105f095>] generic_file_aio_write+0x62/0xb1 (Errors) Jan 10 11:00:09 Tower kernel: [<c1084996>] do_sync_write+0x94/0xcf (Errors) Jan 10 11:00:09 Tower kernel: [<c10cf44c>] reiserfs_file_write+0x67/0x70 (Errors) Jan 10 11:00:09 Tower kernel: [<c10851d2>] vfs_write+0x8a/0xfc (Errors) Jan 10 11:00:09 Tower kernel: [<c10cf3e5>] ? reiserfs_file_open+0x53/0x53 (Errors) Jan 10 11:00:09 Tower kernel: [<c1085287>] sys_pwrite64+0x43/0x5c (Errors) Jan 10 11:00:09 Tower kernel: [<c131fbb5>] syscall_call+0x7/0xb (Errors) Jan 10 11:00:09 Tower kernel: [<c1310000>] ? alloc_node_mem_map+0x70/0x8a (Errors) Jan 10 11:00:09 Tower kernel: Code: 83 f9 10 0f 82 d8 00 00 00 39 fe 0f 82 80 00 00 00 81 f9 a8 02 00 00 72 0c 89 f3 31 fb 81 e3 ff 00 00 00 74 2c 83 e9 10 83 e9 10 <8b> 1e 8b 56 04 89 1f 89 57 04 8b 5e 08 8b 56 0c 89 5f 08 89 57 Jan 10 11:00:09 Tower kernel: EIP: [<c11a95ca>] memmove+0x3b/0x17a SS:ESP 0068:f767b860 Jan 10 11:00:09 Tower kernel: CR2: 00000000f7bfe000 I was already running a memory check - this is fine. Any additional idea?
January 10, 201313 yr That's not a syslog. It says the repeat reiserfsck with the --rebuild-tree option. This should be the next action. repeat reiserfsck with the --rebuild-tree option
January 10, 201313 yr Author I would love to, but the server even doesn't boot anymore. On the entry screen it counts from 5 to 0 and then starting over again. Can the USB flash be the culprit? Can it cause all the issues? I will format it again....unfortunately I don't have a second safety flash.
January 14, 201313 yr Author I have formatted the USB flash. Server is starting again. However, it always comes up with a parity check. I have two options: stop parity check; result: server reboots and is starting with a trace (I can see it on the screen). Afterwards no access at all, not even via the local terminal let parity check running; result: server is becoming unresponsive right after the check Is there any way to stop parity sync via terminal? Any other advise? I'm running into panic mode soon
January 15, 201313 yr Is there any way to stop parity sync via terminal? Yes, at the command prompt type: /root/mdcmd nocheck
January 20, 201313 yr Author It's me again. Memtest is ok. Checkdisk on the flash was ok as well. Output after /root/mdcmd nocheck: root@Tower:~# Message from syslogd@Tower at Sun Jan 20 19:41:51 2013 ... Tower kernel: EIP: [<c10e0577>] is_tree_node+0xe/0x118 SS:ESP 0068:efcada4c Message from syslogd@Tower at Sun Jan 20 19:41:51 2013 ... Tower kernel: Call Trace: Message from syslogd@Tower at Sun Jan 20 19:41:51 2013 ... Tower kernel: Process find (pid: 11019, ti=efcac000 task=f1a3b600 task.ti=efcac000) Message from syslogd@Tower at Sun Jan 20 19:41:51 2013 ... Tower kernel: Code: 39 7d f0 7d 0b 83 c3 18 89 45 ec e9 eb fe ff ff b8 01 00 00 00 83 c4 24 5b 5e 5f 5d c3 55 89 c1 89 e5 57 56 53 83 ec 24 8b 58 18 <0f> b7 03 39 d0 74 31 89 54 24 14 89 44 24 10 c7 44 24 0c 31 0e Message from syslogd@Tower at Sun Jan 20 19:41:51 2013 ... Tower kernel: Stack: Message from syslogd@Tower at Sun Jan 20 19:41:51 2013 ... Tower kernel: EIP: [<c10e10dd>] search_by_key+0x7eb/0xde3 SS:ESP 0068:efe05a84 Message from syslogd@Tower at Sun Jan 20 19:41:51 2013 ... Tower kernel: Call Trace: Message from syslogd@Tower at Sun Jan 20 19:41:51 2013 ... Tower kernel: Process find (pid: 11020, ti=efe04000 task=f1a38000 task.ti=efe04000) Message from syslogd@Tower at Sun Jan 20 19:41:51 2013 ... Tower kernel: Code: 0f b6 d0 0f b7 c2 31 d2 83 fa 00 7f 3a 7c 08 3b 85 5c ff ff ff 77 30 8b 85 58 ff ff ff e8 51 f6 ff ff 8b 86 d0 01 00 00 8b 40 04 <8b> 50 08 c7 85 68 ff ff ff 00 00 00 00 c7 85 38 ff ff ff ff ff Message from syslogd@Tower at Sun Jan 20 19:41:51 2013 ... Tower kernel: Stack: Message from syslogd@Tower at Sun Jan 20 19:41:51 2013 ... Tower kernel: EIP: [<c10e0577>] is_tree_node+0xe/0x118 SS:ESP 0068:efc0fa4c Message from syslogd@Tower at Sun Jan 20 19:41:51 2013 ... Tower kernel: Call Trace: Message from syslogd@Tower at Sun Jan 20 19:41:51 2013 ... Tower kernel: Process find (pid: 11026, ti=efc0e000 task=f1a3b960 task.ti=efc0e000) Message from syslogd@Tower at Sun Jan 20 19:41:51 2013 ... Tower kernel: Code: 39 7d f0 7d 0b 83 c3 18 89 45 ec e9 eb fe ff ff b8 01 00 00 00 83 c4 24 5b 5e 5f 5d c3 55 89 c1 89 e5 57 56 53 83 ec 24 8b 58 18 <0f> b7 03 39 d0 74 31 89 54 24 14 89 44 24 10 c7 44 24 0c 31 0e Message from syslogd@Tower at Sun Jan 20 19:41:51 2013 ... I attach the latest syslog that is on the flash.
January 21, 201313 yr Author I started according to your guidance. After a very short time I see this at the terminal screen: /usr/bin/tail -f /var/log/syslog Jan 21 13:30:25 Tower kernel: [] vfs_write+0x8e/0x110 Jan 21 13:30:25 Tower kernel: [] ? do_sync_write+0xe0/0xe0 Jan 21 13:30:25 Tower kernel: [] ? reiserfs_file_open+0x53/0x53 Jan 21 13:30:25 Tower kernel: [] sys_pwrite64+0x45/0x5c Jan 21 13:30:25 Tower kernel: [] syscall_call+0x7/0xb Jan 21 13:30:25 Tower kernel: [] ? nv_ht_enable_msi_mapping+0x20/0xcb Jan 21 13:30:25 Tower kernel: Code: 83 f9 10 0f 82 d8 00 00 00 39 fe 0f 82 80 00 00 00 81 f9 a8 02 00 00 72 0c 89 f3 31 fb 81 e3 ff 00 00 00 74 2c 83 e9 10 83 e9 10 <8b> 1e 8b 56 04 89 1f 89 57 04 8b 5e 08 8b 56 0c 89 5f 08 89 57 Jan 21 13:30:25 Tower kernel: EIP: [] memmove+0x3b/0x17a SS:ESP 0068:f07eb818 Jan 21 13:30:25 Tower kernel: CR2: 00000000f7bfe000 Jan 21 13:30:25 Tower kernel: ---[ end trace d02e608f9884c747 ]--- Jan 21 13:30:39 Tower afpd[1454]: transmit: Request to dbd daemon (db_dir /mnt/user/Datenaustausch) timed out. Jan 21 13:30:39 Tower afpd[1454]: Reopen volume /mnt/user/Datenaustausch using in memory temporary CNID DB. Reiserfsck was running well at md3 & md4: root@Tower:~# reiserfsck --check /dev/md3 reiserfsck 3.6.21 (2009 www.namesys.com) reiserfsck --check started at Mon Jan 21 13:41:05 2013 ########### Replaying journal: Done. Reiserfs journal '/dev/md3' in blocks [18..8211]: 0 transactions replayed Checking internal tree.. finished Comparing bitmaps..finished Checking Semantic tree: finished No corruptions found There are on the filesystem: Leaves 212436 Internal nodes 1321 Directories 259 Other files 29427 Data block pointers 209086252 (0 of them are zero) Safe links 0 ########### reiserfsck finished at Mon Jan 21 14:28:50 2013 root@Tower:~# reiserfsck --check /dev/md5 reiserfsck 3.6.21 (2009 www.namesys.com) reiserfsck --check started at Mon Jan 21 14:09:16 2013 ########### Replaying journal: Done. Reiserfs journal '/dev/md5' in blocks [18..8211]: 0 transactions replayed Checking internal tree.. finished Comparing bitmaps..finished Checking Semantic tree: finished No corruptions found There are on the filesystem: Leaves 121907 Internal nodes 732 Directories 9766 Other files 18299 Data block pointers 121967818 (0 of them are zero) Safe links 0 ########### reiserfsck finished at Mon Jan 21 14:31:51 2013 Disks 2, 4, 7 are busy: root@Tower:~# umount /dev/md2 umount: /mnt/disk2: device is busy. (In some cases useful info about processes that use the device is found by lsof( or fuser(1)) root@Tower:~# umount /dev/md4 umount: /mnt/disk4: device is busy. (In some cases useful info about processes that use the device is found by lsof( or fuser(1)) root@Tower:~# umount /dev/md7 umount: /mnt/disk7: device is busy. (In some cases useful info about processes that use the device is found by lsof( or fuser(1)) md6 really concerns me (reiserfsck --rebuild-tree /dev/md6): reiserfs_open: the reiserfs superblock cannot be found on /dev/md6. Failed to open the filesystem.
January 21, 201313 yr Author What I forgot to mention: fuser -mv /mnt/disk2 /mnt/user/* is running forever. The same applies for disk 4 & disk 7.
January 25, 201313 yr Author Is it possible to run the ReiserFS check check by disassembling the whole unRAID system, and running the check disk by disk by attaching it to an Ubuntu system? Another advantage would be that I can save all of my files.
January 25, 201313 yr Is it possible to run the ReiserFS check check by disassembling the whole unRAID system, and running the check disk by disk by attaching it to an Ubuntu system? Yes, the file system is on the first (and only) partition on each of the data disks. Another advantage would be that I can save all of my files. If you write to the disks, parity will no longer be valid. In fact, mounting them as writable, but not writing them will still update the superblock introducing some parity errors. When used elsewhere, the only truly safe way to not affect parity is mounted read-only.
January 26, 201313 yr Author After all I bought a new motherboard with 8GB of RAM and a XEON CPU. All the strange behavior is gone but the data errors are still existing. However first success: /dev/md2 which started causing troubles is coming up without errors after a reiserfsck --rebuild-tree: root@Tower:~# reiserfsck --check /dev/md2 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md2 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Sat Jan 26 09:04:25 2013 ########### Replaying journal: Done. Reiserfs journal '/dev/md2' in blocks [18..8211]: 0 transactions replayed Checking internal tree.. finished Comparing bitmaps..finished Checking Semantic tree: finished No corruptions found There are on the filesystem: Leaves 299332 Internal nodes 1811 Directories 3650 Other files 8974 Data block pointers 301956946 (334144 of them are zero) Safe links 0 ########### reiserfsck finished at Sat Jan 26 10:01:43 2013 ########### All of these drives are showing the same encouraging picture: /dev/md3 /dev/md4 /dev/md5 /dev/md7 I think that is good progress. Only two things remains: /dev/md1 root@Tower:~# reiserfsck --check /dev/md1 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md1 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes reiserfs_open: the reiserfs superblock cannot be found on /dev/md1. Failed to open the filesystem. If the partition table has not been changed, and the partition is valid and it really contains a reiserfs partition, then the superblock is corrupted and you need to run this utility with --rebuild-sb. /dev/md6 root@Tower:~# reiserfsck --check /dev/md6 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md6 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes reiserfs_open: the reiserfs superblock cannot be found on /dev/md6. Failed to open the filesystem. If the partition table has not been changed, and the partition is valid and it really contains a reiserfs partition, then the superblock is corrupted and you need to run this utility with --rebuild-sb. Both disks are shown as "UNFORMATTED". Please advise how to move forward. Thanks a lot.
January 28, 201313 yr Author I found some advise and here are the steps I was taking: reiserfsck --rebuild-sb /dev/md1 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will check superblock and rebuild it if needed Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes reiserfs_open: the reiserfs superblock cannot be found on /dev/md1. what the version of ReiserFS do you use[1-4] (1) 3.6.x (2) >=3.5.9 (introduced in the middle of 1999) (if you use linux 2.2, choose this one) (3) < 3.5.9 converted to new format (don't choose if unsure) (4) < 3.5.9 (this is very old format, don't choose if unsure) (X) exit 1 Enter block size [4096]: 4096 No journal device was specified. (If journal is not available, re-run with --no-journal-available option specified). Is journal default? (y/n)[y]: Did you use resizer(y/n)[n]: rebuild-sb: no uuid found, a new uuid was generated (ff945f76-fadc-47e6-92c9-193982656fdd) rebuild-sb: You either have a corrupted journal or have just changed the start of the partition with some partition table editor. If you are sure that the start of the partition is ok, rebuild the journal header. Do you want to rebuild the journal header? (y/n)[n]: y Reiserfs super block in block 16 on 0x901 of format 3.6 with standard journal Count of blocks on the device: 366284624 Number of bitmaps: 11179 Blocksize: 4096 Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 0 Root block: 0 Filesystem is NOT clean Tree height: 0 Hash function used to sort names: not set Objectid map size 0, max 972 Journal parameters: Device [0x0] Magic [0x0] Size 8193 blocks (including 1 for journal header) (first block 18) Max transaction length 1024 blocks Max batch size 900 blocks Max commit age 30 Blocks reserved by journal: 0 Fs state field: 0x1: some corruptions exist. sb_version: 2 inode generation number: 0 UUID: ff945f76-fadc-47e6-92c9-193982656fdd LABEL: Set flags in SB: Mount count: 1 Maximum mount count: 30 Last fsck run: Mon Jan 28 17:20:08 2013 Check interval in days: 180 Is this ok ? (y/n)[n]: y The fs may still be unconsistent. Run reiserfsck --check. root@Tower:/# reiserfsck --check /dev/md1 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md1 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Mon Jan 28 17:21:24 2013 ########### Replaying journal: Trans replayed: mountid 635, transid 6473107, desc 5646, len 1, commit 5648, next trans offset 5631 Trans replayed: mountid 635, transid 6473108, desc 5649, len 1, commit 5651, next trans offset 5634 Trans replayed: mountid 635, transid 6473109, desc 5652, len 1, commit 5654, next trans offset 5637 Trans replayed: mountid 635, transid 6473110, desc 5655, len 7, commit 5663, next trans offset 5646 Trans replayed: mountid 635, transid 6473111, desc 5664, len 7, commit 5672, next trans offset 5655 Replaying journal: Done. Reiserfs journal '/dev/md1' in blocks [18..8211]: 5 transactions replayed Checking internal tree.. \/ 7 (of 10-/ 33 (of 162\/ 83 (of 170-bad_indirect_item: block 365558867: The item [42063 1941647 0x47a001 IND (1)] has the bad pointer (751) to the block (366284624) bad_indirect_item: block 365558867: The item [42063 1941647 0x47a001 IND (1)] has the bad pointer (752) to the block (366284625) bad_indirect_item: block 365558867: The item [42063 1941647 0x47a001 IND (1)] has the bad pointer (753) to the block (366284626) bad_indirect_item: block 365558867: The item [42063 1941647 0x47a001 IND (1)] has the bad pointer (754) to the block (366284627) bad_indirect_item: block 365558867: The item [42063 1941647 0x47a001 IND (1)] has the bad pointer (755) to the block (366284628) bad_indirect_item: block 365558867: The item [42063 1941647 0x47a001 IND (1)] has the bad pointer (756) to the block (366284629) bad_indirect_item: block 365558867: The item [42063 1941647 0x47a001 IND (1)] has the bad pointer (757) to the block (366284630) bad_indirect_item: block 365558867: The item [42063 1941647 0x47a001 IND (1)] has the bad pointer (758) to the block (366284631) bad_indirect_item: block 365558867: The item [42063 1941647 0x47a001 IND (1)] has the bad pointer (759) to the block (366284632) bad_indirect_item: block 365558867: The item [42063 1941647 0x47a001 IND (1)] has the bad pointer (760) to the block (366284633) bad_indirect_item: block 365558867: The item [42063 1941647 0x47a001 IND (1)] has the bad pointer (761) to the block (366284634) bad_indirect_item: block 365558867: The item [42063 1941647 0x47a001 IND (1)] has the bad pointer (762) to the block (366284635) bad_indirect_item: block 365558867: The item [42063 1941647 0x47a001 IND (1)] has the bad pointer (763) to the block (366284636) bad_indirect_item: block 365558867: The item [42063 1941647 0x47a001 IND (1)] has the bad pointer (764) to the block (366284637) finished Comparing bitmaps..finished Checking Semantic tree: finished 14 found corruptions can be fixed when running with --fix-fixable ########### reiserfsck finished at Mon Jan 28 18:15:01 2013 ########### Finally a reiserfsck --fix-fixable /dev/md1 repaired the remaining 14 corruptions. Done with /dev/md1 /dev/md6 was more difficult. After a reiserfsck --rebuild-sb /dev/md6 and a reiserfsck --check /dev/md6 (based on the message: The fs may still be unconsistent. Run reiserfsck --check) it completed with a Bad root block 0. (--rebuild-tree did not complete) A reiserfsck --rebuild-tree /dev/md6 was needed to rebuild /dev/md6. Thanks to dgaschk and Joe L. for their help, advise and patience.
Archived
This topic is now archived and is closed to further replies.