Rebuild goes wrong

Everything looked nominal. This case is actually kind of unexpected example of routine operation going belly up, the NAS owner ends up looking for recovery options after RAID 5 crash

Recently our quite old NAS Iomega Storcenter 200rl with a RAID 5 out of 4x500GB, reported a harddisk failure and our administration replaced one of the disks that was marked. Then the Storecenter told that it was synchronizing the data. After synchronization the NAS told that 100% of storage capacity is free and so it tell until now… I am interessted in options to revocer the data.

Certainly does not look like something was done wrong; if the NAS have sensed the rebuild cannot be completed, it would have had refused to even start the process. Certainly, with a wrong disk replaced, the rebuild does not happen. In this case, however, the rebuild was completed with no reported anomaly – something else went wrong.

With something unknown going wrong, it is difficult to predict if the case is recoverable. One can give our Home NAS Recovery a spin, but there is no guarantee of success. Cases when rebuild goes wrong for no apparent reason are always dicey.

 

 

Network Recycle Bin and undelete

This specific question is about IX4-300d, but it applies to virtually any NAS out there.

The question is about some Recycle Bin equivalent for a NAS.

As in most NAS (if not all) when a file is deleted from a share in IX4-300d it disaappear “completely” from the drive. No chances to recover it from a Recycle bin. I read in a post that UNDELETE by Diskeeper might be a solution creating a “virtual” recycle bin. I decided to try it and downloaded a trail version of the program. But it did not recognized the drive/shares in the NAS. Does anyone has any experience with UNDELETE in NAS?

First of all, some NAS vendors provide an equivalent of the Windows Recycle Bin. QNAP, for example, calls this feature Network Recycle Bin. It keeps files retrievable as best practically possible (subject to age and free space constraints).

Whatever delete protection mechanisms, they must be implemented and run on the NAS side, because it is the NAS filesystem that owns the files and performs deletions. Any Windows-based installable recycle bin software only works with locally attached disks.

After the files are deleted with no Recycle Bin in place, there is still a chance – our Home NAS Recovery can help (with varying success depending on how long since the deletion).

Prevention is good

Every now and then, people plan and go to great lengths figuring how to do data recovery before the actual data loss occurs.

What type of file system does the ix2 use with hard drives? For safety, I’d like to know how to recover data from the drives if the ix2 fails. I do not like the idea of relying on some hidden proprietary format that requires the ix2 to be able to retrieve my data in case of failure.

Anyone know how to read the disk formatted by the ix2 from a PC running any operating system?

There is no hidden proprietary format actually. Lenovo/Iomega uses run-of-the-mill Linux software and filesystems. However, the great confusion ensues little further in the discussion, mixing up filesystems with RAIDs, RAIDs with backup, single-disk RAID1 with a single partition, you name it.

Well, let’s spell it out

  1. Any NAS would use this-or-that form of RAID.
    1. Even if you have a single disk in a multi-bay NAS, it will have a single-disk RAID, most likely RAID1, to avoid changing data format if you add a second drive.
    2. Furthermore, if you have a single-bay NAS, it is still likely to have a single-disk RAID, to achieve compatibility across the entire model lineup (and specifically the ability to move the drive to the multi-bay model).
  2. Most modern NASes use one of the standard Linux filesystems, XFS, EXT3, EXT4, or BTRFS.
  3. If there is no damage to the data on disks, or the damage is minor, and assuming skilled operator, a properly configured Linux PC will read the data if all disks are connected at the same time.

But, the most important question here is: if one plans for data recovery before the actual loss, why do not have a backup instead of recovery?

 

ix4-200d unsuccessful drive replacement

Someone had to go to a data recovery center when his Iomega StorCenter ix4-200d has a rebuild failure

I have a StoreCenter ix4-200d [with 4 disks in RAID10] that had a drive failure. I … replaced the damaged drive. On restart the screen on the StoreCenter said “Drive 1 requires overwrite confirmation“, but when I go into the dashboard, there is no message for confirmation, and it has not lost all storage folders and shows 0 data stored.

… and that’s actually the end of story. Further troubleshooting attempts in earnest require the drives to be out of the NAS, no matter if you use Linux and mdadm, or Windows and data recovery software. However, as it often happens, some tinkering with mdadm was attempted, unfortunately with no effect, but fortunately without any ill side effects. The story ends happily, although not cheap

 

Yes i found a solution… I did send my storecentre to a recovery service centre and they managed to restore the data 100%…the total cost for this recovery €3000 !!!

That may seem tad high for a 4 disks in RAID10, as the drives were still readable and probably did not both require extensive mechanical work. The corresponding Home NAS Recovery license would have been little short of $200, plus a cost of four replacement drives.

IX4 falling apart

This documents a typical sequence of multiple drives failing in a RAID5, this time in Iomega ix4-200d.

I’ve received an automatic email from the dashboard saying

Data protection is being reconstructed. Data is available during this operation, however performance may be degraded.

After that, the NAS started ‘Data recovery procedure’ [and then came another message]

Drive number 1 encountered a recoverable error.

[and] NAS started recovery procedure from the scratch. Even though that mentioned drive has failed, everything worked fine untill yesterday [when] new message … said

Storage failed and some data loss may have occurred. Multiple drives may have either failed or been removed from your storage system. Visit the Dashboard on the management interface for details.

Is there any way to recover at least some data if it is NAS that failed?

Depending on the condition of the disks, Home NAS Recovery may or may not be able to extract data from it.

Maybe there was a spare involved, as data protected is being reconstructed in a RAID5 can only refer to a rebuild of the array. The rebuild happens either when a defective disk was replaced or when a hot spare kicks in after one of the active array disks fails. There is another variation to that tune, not really obvious, which is a transient failure causing one of the disks to drop out of the array momentarily, then report back online and be accepted back in the array.

Anyhow, while rebuild is in progress, it turns out the second drive in the array is unreadable. This halts the rebuild. The error is at first deemed recoverable, and the rebuild is retried. However, the error recovery is not successful and the second disk (#1) drops offline. With two drives offline, the data is no longer accessible.

For our data recovery software, the logical reconstruction is not a problem given that the disks are still readable enough. This may be a problem though. In worst case, the disks need to be cloned to a blank new disks, and clones then used for recovery. The NAS will not accept the clones because it already has recorded the disks as “failed”, and cloning the entire disk content also clones the “failed” marks for respective disks.

Iomega StorCenter ix2-200 device fails when powered

This is about an Iomega/Lenovo device, Iomega StorCenter ix2-200

When I power on the ix2 the ‘!’ light blinks red. I can’t connect to it…, under ‘dashboard’ everything seems as usual except the pie chart showing space usage is not there. … when I click on Users or Shared Storage it says Disks Not Ready. The selected function is not available due to the state of the disks.

… what to do from here?

Although it is not specified anywhere in the post, further discussion suggests there are two disks in the NAS. Another crucial bit of information missing is the RAID level. There are three possibilities with two disks:

  • RAID1, when two disks are identical;
  • JBOD, when the data is first stored on one disk, then once the first disk fills up, the second one is used;
  • RAID0, when the data is interleaved between two disks.

Red exclamation sign on a StorCenter indicates either a non-recoverable disk failure, or some kind of severe logical failure.

First thing to try, as rightly suggested in follow-ups to the original post, is to try booting with just one disk. This works if the array is RAID1. Two tests must be done because even with RAID1, there is an even chance of leaving the bad disk in.

If the array is RAID0 or JBOD, and the drive ha indeed failed, the drive must be repaired first. If there is no mechanical problem, but rather some logical issue, we can help you with RAID0, but not with a JBOD.

 

Lenovo PX6-300D

This describes behavior of the PX6 6-bay Lenovo NAS with multiple disk failures,

I have px6-300D nas with 3TB X 6 drives. I configured it with Raid 5. Few Days back it was showing a message The amount of free space on your ‘Shares’ volume is below 5% of capacity. and asked to overwrite Drive 6…Then i contacted customer care they told that your few drives (3 or 4) has failed. … and go with some data recovery solution provide… If its NAS with raid protection my data must be protected. I really need my data back.

RAID protection is great but it has its limits. It does not protect against anything else than disk failure, and RAID5 only protects against a single disk failure. Multiple disks fail, down it goes.

Reconstucting restarts at 45% starts with 0.

That’s what it looks like when implemented by Lenovo. Other vendors will have different indications, but the end result is the same and the array cannot be rebuilt. Short of packing the disks for a data recovery service, what else can be done?

  1. Cheapest option is to remove all the disks from the NAS, clone them to a set of new disks of the same capacity, and put the clones back. The NAS will hopefully pick up the copies and completes the rebuild successfully.
  2. If the rebuild does not pick up, our Home NAS Recovery software can in all likelihood do the job.