July 23, 201015 yr I just got what I believe is all the equipment I need to build my first unraid server yesterday. Most of the equipment comes from somewhat dated computer components I have had laying around. Everything is built and I can access the console across the LAN just fine. I don't think I thought this through very well, but the first thing I did was assign the parity disk and data disk (I have only two 2TB drives to start). I then assumed I would need to format the drive so chose to do this from the console. It has now been churning overnight and should be complete in the next 2 hours or so. My question is, since I have plans to transfer over 1TB of data to this server have I made an inefficient step by assigning the parity drive this early? I think I read somewhere that you should not assign a parity drive until after you have completed the batch transfer of data you are hoping to move to the server. By not assigning the parity drive you allow yourself to have the maximum transfer rate and since you already have a backup of the data recovery protection isn't really a concern. So I guess I'm confused about what I did. I thought I was only formatting the drives, but could it be performing a format and a parity check at the same time? I've attached a screenshot of what my console displays at the moment if that helps at all. Ultimately, could someone walk me through the steps of what most do when they just get their unraid server online? Essentially, what disks do you assign and when, when do you format, and when do you parity sync? My assumption is you assign the data drive first, format the data drive, complete the transfer, then assign the parity disk, then do a paarity sync, and you are done. Obviously I didn't do this, but am I right? So whether I've done the right thing or not can anyone tell me what I should do from here? After the server is done doing what its doing, which again, I can't say I fully understand is it formatting, a parity sync, or both? Should I simply unassign the parity disk and then complete the transfer and assign it back? I'm just generatlly confused but I think the source of it for me right now is I can't fully understand what the console is attempting to reflect.
July 23, 201015 yr It is currently doing an initial parity calculation based on the contents of the data disk. The data disk has not yet been formatted, and will not be until you check the box under the "Format" button and then press it. You can do that at any time, including now as it is calculating parity. Typically, it takes 30 seconds or so to format a drive, although I've seen it take up to a few minutes with large drives and slower hardware. So, check the box, press format. You might as well un-assign the parity drive and then, at the linux command line, after logging in as "root" type: initconfig This will set a new initial configuration based on the existing assigned and working disks. The "initconfig" command also immediately invalidates any prior parity calculation as a new set of parity calculations will be generated when you next assign the parity drive and start the array. When you then go back to the web-management screen and refresh it you'll see the data disk as "green" ("initconfig" does not delete any data on data disks, they are un-affected) Transfer your files at that point. Once you have the initial set of files transferred, stop the array, go to the devices page, assign the parity drive, go back to the main page, and press "Start" once more. This time an initial parity calculation will take place. Once it is done you'll have parity protection. Joe L.
July 23, 201015 yr Author First, Joe L., thank you so much for always helping. I've only made a few posts here and on the avsforum and in each case you have really helped me. Thanks again. I actually have checked the box and pressed format now twice (early on in the parity check). I'm not entirely sure why it still reads as "unformatted." I guess I'll just wait for the parity check to finish and see if it indeed reflects that it has been formatted. It evidently has another hour. Thanks for passing on the steps to get me where I need to go from here. Basically, it sounds like I need to start over since it now appears to me it is the parity check that takes the most time (excluding the data transfer). I now think I understand what the better process is: 1. upon first viewing the web management screen assign only your data discs 2. format the drives 3. transfer the data 4. assign the parity disc 5. complete the parity sync It sounds like the initconfig command puts me back to the initial state and pretty much allows me to follow steps 1 - 5 above. I guess all I'm doing now is the parity sync of a blank disc...correct? Finally, two more quick questions if you don't mind. - Is there any way to send the initconfig command to the unraid server from one of the windows clients? This isn't a big deal and really more of a curiosity than anything. I'm just not at home right now and can only remote in to the windows clients and perform commands from there. This would only save me some time since I could start the data transfer before I get home. What about just unassigning the parity disc from the web management screen? That won't do it? - This unraid server will be used as a media server. I apologize for my ignorance here, but I'm wondering about shares as they relate to expanding the array. I know that I can setup a single share that will allow me to use the space of the current data disc (only one at the moment), but what happens when I add another drive? Will the same share see the new available disc space or do you need to create another one? I'm only wondering because after I transfer 1TB of data to the current unraid server I want to take that same 1TB drive and add it to the unraid server. I guess I could wait to create the share until all the data drives are present, but I guess I'm curious how the shares work when adding new drives to the array since I know I'll be met with that in the future.
July 23, 201015 yr First, Joe L., thank you so much for always helping. I've only made a few posts here and on the avsforum and in each case you have really helped me. Thanks again. I actually have checked the box and pressed format now twice (early on in the parity check). I'm not entirely sure why it still reads as "unformatted." I guess I'll just wait for the parity check to finish and see if it indeed reflects that it has been formatted. It evidently has another hour. Thanks for passing on the steps to get me where I need to go from here. Basically, it sounds like I need to start over since it now appears to me it is the parity check that takes the most time (excluding the data transfer). I now think I understand what the better process is: 1. upon first viewing the web management screen assign only your data discs 2. format the drives 3. transfer the data 4. assign the parity disc 5. complete the parity sync It sounds like the initconfig command puts me back to the initial state and pretty much allows me to follow steps 1 - 5 above. I guess all I'm doing now is the parity sync of a blank disc...correct? Correct. It is calculating parity on whatever is on the first partition of the data disk. If unformatted, it is all zeros, if formatted, then some bytes are non-zero, set that way by the basic format structures of a reiser file-system. Finally, two more quick questions if you don't mind. - Is there any way to send the initconfig command to the unraid server from one of the windows clients? This isn't a big deal and really more of a curiosity than anything. I'm just not at home right now and can only remote in to the windows clients and perform commands from there. This would only save me some time since I could start the data transfer before I get home. What about just unassigning the parity disc from the web management screen? That won't do it? Yes, there is. On the management console, press "Stop" to stop the array. (the array must be stopped first) Then enter the following URL in your browser: //your_server_name/blank.htm?cmdInit=apply It is exactly what the initconfig command will do if you were to log in via the command line. - This unraid server will be used as a media server. I apologize for my ignorance here, but I'm wondering about shares as they relate to expanding the array. I know that I can setup a single share that will allow me to use the space of the current data disc (only one at the moment), but what happens when I add another drive? Will the same share see the new available disc space or do you need to create another one? I'm only wondering because after I transfer 1TB of data to the current unraid server I want to take that same 1TB drive and add it to the unraid server. I guess I could wait to create the share until all the data drives are present, but I guess I'm curious how the shares work when adding new drives to the array since I know I'll be met with that in the future. All parallel - like-named directories on each of the data disks will be presented as a single "merged" view as a user-share. Just plan on a few top level directories/folders and put your media in them. I use "Movies," "Pictures", "Mp3," and "data" as the top level folders. Under them I have sub-folders where needed. After you transfer 1TB to the server you'll want to assign the parity drive before re-use of the original drive as an unRAID data drive. That way you have some protection from a disk failure. The odds of a failure in the early hours of a disk's use are much higher than after it is proven mechanically. Joe L.
July 23, 201015 yr Author Thanks again. The parity sync just completed, but I noticed the data drive still reflected "Unformatted." I did check the checkbox and clicked "Format" early on in the parity sync process twice. I also did it now, but the data disk continues to reflect "Unformatted." I have naturally "Refresh"-ed as well. I think in an earlier post you mentioned this wouldn't take very long. Any chance this makes any sense?
July 23, 201015 yr Thanks again. The parity sync just completed, but I noticed the data drive still reflected "Unformatted." I did check the checkbox and clicked "Format" early on in the parity sync process twice. I also did it now, but the data disk continues to reflect "Unformatted." I have naturally "Refresh"-ed as well. I think in an earlier post you mentioned this wouldn't take very long. Any chance this makes any sense? What version of unRAID are you running? regardless, for more analysis, you'll need to attach a syslog to your next post. Instructions on how to capture it are here in the wiki: http://lime-technology.com/wiki/index.php?title=Troubleshooting#Capturing_your_syslog Joe L.
July 23, 201015 yr It's 4.5.6 Joe so it should not be showing a formatted disk as unformatted. However, maybe there is still a bug of some sort. Try to stop and then start the array. That should "reset" it. The format should only take a few moments. With a new build, unRAID just creates the parity to match the data disks. When you add a new disk to an existing array, the new disk has to be cleared to be all 0's so it does not break the parity and that takes a while. In the shares, you can tell unRAID which disks to use for each share. So, you just include or exclude the disks you want and I think blank means all disks. You can also just go to "disk1" and create the top level directorys you want and begin to move files. Just turn-on user shares and those top-level directories will appear as shares by themselves (I think only after a reboot). Writing to shares can be tricky to ensure the data for a common item, like the parts of a ripped movie or the songs of a certain artist, stay on the same disk which makes access and maintenance easier. But writing to disks and just reading the shares is easy to use. If writing to the disks, then once you have a disk2 just go to it and create an exact same top level directory and then both of those directories get combined as one share when you view the share. My main shares are Movies, TV Shows and Music. I have them limited to different disks. Actually, with only one disk I would expect that unassigning the parity disk without a "restore" would work the same as only having the data disk. Peter
July 23, 201015 yr It's 4.5.6 Joe so it should not be showing a formatted disk as unformatted. However, maybe there is still a bug of some sort. Could be, that's why I asked he supply a syslog for analysis. (after pressing the "format" button, but before rebooting) Actually, with only one disk I would expect that unassigning the parity disk without a "restore" would work the same as only having the data disk. Peter Actually, un-assigning any drive is treated identically to a failed drive unless you set a new initial disk configuration. Therefore, to unRAID, un-assigning the parity drive (without an "initconfig" which will cause it to forget the drive exists in the array) is identical to having the parity drive fail. Joe L.
July 23, 201015 yr Author Thanks for the additional help. This really has me stumped. As lionelhutz mentioned I'm using 4.5.6. I tried using the initconfig command which appears to have gotten me to a clean slate. I then assigned only the WD disc as a data disc (the one which doesn't apparently want to format) and then attempted to format again. It initially says "formatting" but if I wait a minute or two and then "Refresh" it continues to reflect as "Unformatted." I think I've managed to extract the syslog. It is attached. The time/date stamps seem a little off to me and that may be from the fact that I didn't have the Central Time Zone checked off yet. I've done that now. 2010_07_23-syslog.txt
July 23, 201015 yr The syslog clearly shows what is happening: Jul 22 19:11:29 Tower kernel: 223MB LOWMEM available. Jul 23 09:09:54 Tower logger: mkreiserfs 3.6.21 (2009 www.namesys.com) Jul 23 09:09:54 Tower logger: Jul 23 09:09:54 Tower kernel: mkreiserfs invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0 Jul 23 09:09:54 Tower kernel: Pid: 2072, comm: mkreiserfs Not tainted 2.6.32.9-unRAID #5 Jul 23 09:09:54 Tower kernel: Call Trace: Jul 23 09:09:54 Tower kernel: [<c104ab61>] oom_kill_process+0x59/0x1cd Jul 23 09:09:54 Tower kernel: [<c104afb9>] __out_of_memory+0xef/0x102 Jul 23 09:09:54 Tower kernel: [<c104b02a>] out_of_memory+0x5e/0x83 Jul 23 09:09:54 Tower kernel: [<c104cfe9>] __alloc_pages_nodemask+0x375/0x42f Jul 23 09:09:54 Tower kernel: [<c1059686>] handle_mm_fault+0x254/0x8f1 Jul 23 09:09:54 Tower kernel: [<c12312aa>] ? net_rx_action+0x57/0x102 Jul 23 09:09:54 Tower kernel: [<c10282cd>] ? __do_softirq+0xf0/0xf8 Jul 23 09:09:54 Tower kernel: [<c1017050>] do_page_fault+0x17c/0x1e4 Jul 23 09:09:54 Tower kernel: [<c1016ed4>] ? do_page_fault+0x0/0x1e4 Jul 23 09:09:54 Tower kernel: [<c12a07ce>] error_code+0x66/0x6c Jul 23 09:09:54 Tower kernel: [<c1016ed4>] ? do_page_fault+0x0/0x1e4 Jul 23 09:09:54 Tower kernel: Mem-Info: Jul 23 09:09:54 Tower kernel: DMA per-cpu: Jul 23 09:09:54 Tower kernel: CPU 0: hi: 0, btch: 1 usd: 0 Jul 23 09:09:54 Tower kernel: Normal per-cpu: Jul 23 09:09:54 Tower kernel: CPU 0: hi: 90, btch: 15 usd: 22 Jul 23 09:09:54 Tower kernel: active_anon:12139 inactive_anon:895 isolated_anon:0 Jul 23 09:09:54 Tower kernel: active_file:20 inactive_file:0 isolated_file:0 Jul 23 09:09:54 Tower kernel: unevictable:36737 dirty:0 writeback:0 unstable:0 Jul 23 09:09:54 Tower kernel: free:667 slab_reclaimable:524 slab_unreclaimable:1513 Jul 23 09:09:54 Tower kernel: mapped:1195 shmem:16 pagetables:131 bounce:0 Jul 23 09:09:54 Tower kernel: DMA free:948kB min:132kB low:164kB high:196kB active_anon:14912kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15804kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:32kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jul 23 09:09:54 Tower kernel: lowmem_reserve[]: 0 206 206 206 Jul 23 09:09:54 Tower kernel: Normal free:1720kB min:1768kB low:2208kB high:2652kB active_anon:33644kB inactive_anon:3580kB active_file:80kB inactive_file:0kB unevictable:146948kB isolated(anon):0kB isolated(file):0kB present:211136kB mlocked:0kB dirty:0kB writeback:0kB mapped:4780kB shmem:64kB slab_reclaimable:2096kB slab_unreclaimable:6052kB kernel_stack:488kB pagetables:492kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:442 all_unreclaimable? yes Jul 23 09:09:54 Tower kernel: lowmem_reserve[]: 0 0 0 0 Jul 23 09:09:54 Tower kernel: DMA: 1*4kB 0*8kB 1*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 948kB Jul 23 09:09:54 Tower kernel: Normal: 0*4kB 1*8kB 1*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1720kB Jul 23 09:09:54 Tower kernel: 36775 total pagecache pages Jul 23 09:09:54 Tower kernel: 0 pages in swap cache Jul 23 09:09:54 Tower kernel: Swap cache stats: add 0, delete 0, find 0/0 Jul 23 09:09:54 Tower kernel: Free swap = 0kB Jul 23 09:09:54 Tower kernel: Total swap = 0kB Jul 23 09:09:54 Tower kernel: 57280 pages RAM Jul 23 09:09:54 Tower kernel: 0 pages HighMem Jul 23 09:09:54 Tower kernel: 1704 pages reserved Jul 23 09:09:54 Tower kernel: 3880 pages shared Jul 23 09:09:54 Tower kernel: 53504 pages non-shared Jul 23 09:09:54 Tower kernel: Out of memory: kill process 2072 (mkreiserfs) score 959 or a child Jul 23 09:09:54 Tower kernel: Killed process 2072 (mkreiserfs) You do not have enough memory to run the process that formats the disk. Joe L.
July 23, 201015 yr Author I can't say I understand the log files very well, it is mostly greek to me, but I did just notice the "out of memory" error. I feel awful now, but to be honest I didn't think memory would affect just simply formatting a drive. You see, before I bought the 2TB drives I created a test setup using some smaller IDE drives. They formatted just fine using the same older components I now have in this server. I even made a post asking about memory and received responses that 512 megs was fine, but that memory really didn't help performance all that much. Ultimately, since I had a test setup that worked with 256 megs I thought I would at least see what the transfer rates (performance) were with the new larger drives before purchasing more memory. Again, thanks for the help and I apologize for the confusion. I'll be purchasing a RAM upgrade today.
July 23, 201015 yr I can't say I understand the log files very well, it is mostly greek to me, but I did just notice the "out of memory" error. I feel awful now, but to be honest I didn't think memory would affect just simply formatting a drive. You see, before I bought the 2TB drives I created a test setup using some smaller IDE drives. They formatted just fine using the same older components I now have in this server. I even made a post asking about memory and received responses that 512 megs was fine, but that memory really didn't help performance all that much. Ultimately, since I had a test setup that worked with 256 megs I thought I would at least see what the transfer rates (performance) were with the new larger drives before purchasing more memory. Again, thanks for the help and I apologize for the confusion. I'll be purchasing a RAM upgrade today. It is simply that formatting the larger disk takes more memory. Here's a possible work-around. (might work, might not, but worth it to see if you are right on the edge of having enough memory) From the unRAID console, stop the array and then disable user shares. Next, log in and teminate "samba" by typing /root/samba stop Then, terminate the management interface program by typing: killall emhttp These will all free up some potential memory for the mkreiserfs program. Next, type: mkreiserfs /dev/md1 If it completes without an error, great. Type reboot to reboot. (or press your reset button on the server, or power cycle the server) If an error still occurs, then you'll definitely need the extra memory. (and you still need to reboot to re-start everything) Joe
July 23, 201015 yr Author Thanks Joe. The steps did work, but the disc still won't format. At the very least this has all been a learning experience for me. Thanks again for taking your time.
July 23, 201015 yr Looks like you just need more memory then. 256 Meg is not enough to format a 2TB disk. Joe L.
July 24, 201015 yr Author Just putting some closure to this topic. I installed an additional gig of RAM and the formatting worked without a hitch. I'm not quite sure if the transfer rates are very good with this build since I am getting 37 meg/second without the parity disc assigned, but I need to do more research. Thanks again for all the help.
August 12, 201015 yr This topic seems to be somewhat related to a question i had about user shares spanning across different discs. Hoping to get some clarification. I have just 2 data discs. Created user shares for movies split level 2. I then created folders for movies/blu ray and movies/dvd. Started flling array. I see the folder struture on disk 1 but not on disc 2. Is it my understanding that you have to manually create the same folder structure for each disk added for unraid to span contents, or are the folders created automatically as needed? Disk 1 is 2tb @61% full, and disc 2 160gb at 0% full. Thanks!!!!
August 12, 201015 yr This topic seems to be related to a question i had about user shares spanning across different discs. Hoping to get some clarification. I have just 2 data discs. Created user shares for movies. i then created folders for movies/blu ray and movies/dvd. Started flling array. I see the folder struture on disk 1 but not on disc 2. Is it my understanding that you have to manually create the same folder structure for each disk added for unraid to span contents, or are the folders created automatically as needed? Disk 1 is 2tb @61% full, and disc 2 160gb at 0% full. Thanks!!!! Any top-level folders on different disks that unRAID finds will automatically become user shares (and they will show up on your Shares page of the web interface). So in your case, if you added a new data disk (disk 3) and created a folder called 'Movies' on it, unRAID would automatically combine the contents of that folder with your 'Movies' user share. Alternately, you can create a user share on the Shares page of the web interface and let unRAID take care of creating the actual folders. Which disks unRAID chooses to use will depend on the settings you specify for that share, specifically the following settings: split level included/excluded disks disk allocation method In your case, 'Movies' is your top-level folder, hence it is your user share. Your 'Blu-Ray' and 'DVD' folders are inside that user share, and are therefore limited by the user share's settings. Any of the above three settings could be preventing unRAID from using disk2. If I had to guess, it is probably your split level, it may need to be lowered. Min free space can also prevent a disk from being used if it is set too high (I recommend leaving it at the default 2 GB). If you post your specific user share settings we can advise you further. Oh, and by the way, your question has nothing to do with this topic. But I won't hold that against you.
August 12, 201015 yr Sorry about the inadvertent hijacking. Should have created new topic for this. All of my movies are in folder format so that is why i chose split level 2. (Example: movies/dvd/Spirited Away (2001)/video_ts.....etc) I want the actual movie and its folder contents to remain intact but span different discs. I left "free space" blank,"allocation" as high water, and "included discs" as blank. As i add each movie folder, I was hoping that this user share setting would enable unraid to automatically duplicate the movie/dvd folder structure onto different discs it sees fit. If i have to make each folder structure manually that's fine too, although i know this would be a pain when you get 10+discs installed into the arrray.
August 12, 201015 yr Split level is a confusing topic, but as I understand it, it is basically asking 'After what level is it OK to split up files?' Meaning take your split level setting +1 and you get the level that unRAID will actually start splitting up files. I believe your 'Movies' share would be considered level 1, then your DVD folder level 2. So I think unRAID isn't using disk2 because it thinks it cannot split up your DVD folder. I believe changing your split level to 3 will solve your problem. This will allow individual movie folders (Sprited Away) to live on different disks, and both the Movies and DVD folders will be automatically recreated on different disks. Everything inside a specific movie folder (Spirited Away's Video_ts etc) will reside on one disk only. I'm not 100% certain about this. I would recommend trying split level 3 and then starting a new transfer into your Movies share. See where it goes. If it goes to disk2, great, problem solved. If it goes to disk1, then cancel the transfer, try split level 4, and try again. Keep playing around with this until you find the right level. By the way, changing your split level will not affect the data already on your server, it will only affect new transfers. So you can change it all you want and only affect future events - unRAID won't start moving your already in place data.
August 12, 201015 yr Yes, unRAID will create the directory structure on each but only as the space is required. You do not have to manually create them. Your structure does not need a split level of 3. Level 1 means that the Movies share can be created on each disk. Level 2 means that the Movies share and any directory under the Movies share can be created on each disk. Your disks have not split because you are likely using high water. High water won't move to disk2 until disk1 is mostly full. The highwater goes something like this; 1st HW level HW level = 1TB disk 1 - 2TB free > 1TB free disk 2 - 160gig free < 1TB free system starts to use disk1 2nd HW level - HW level is reset when free space on all disks is < 1st HW level HW level - 500gig disk 1 - 1TB free > 500gig disk 2 - 160gig free < 500gig system continues to use disk1 3nd HW level - HW level is reset when free space on all disks is < 2nd HW level HW level - 250gig disk 1 - 500gig free > 250gig disk 2 - 160gig free < 250gig system continues to use disk1 4th HW level - HW level is reset when free space on all disks is < 3rd HW level HW level - 125gig disk 1 - 250gig free > 125gig disk 2 - 160gig free > 125gig System now uses disk2 since has more free space than the high level yet it has the least free space. So, the conclusion is that your server will not move to disk2 until disk1 has less than 250gig of free space or is >87.5% full. Read the high water description part way down here; http://www.lime-technology.com/wiki/index.php?title=UnRAID_Manual#User_shares_2 Split level is a confusing topic, but as I understand it, it is basically asking 'After what level is it OK to split up files?' Meaning take your split level setting +1 and you get the level that unRAID will actually start splitting up files. This is not a correct definition. The split level is the number of directory levels down into the share which are allowed to be created on multiple disks. 1 = Share can split 2 = Share and subdirectories under the share can split. Peter
August 12, 201015 yr thanks everyone for your points. It certainly clarifies my understanding on the matter of split levels. I will continue to fill up drive and see how things go.
Archived
This topic is now archived and is closed to further replies.