The Truth About Securely Wiping Data

[+Xinok Subscriber²]

Sources:

http://blogs.computerworld.com/node/5687

http://www.nber.org/sys-admin/overwritten-data-gutmann.html

http://en.wikipedia.org/wiki/Data_recovery...verwritten_data

http://en.wikipedia.org/wiki/Gutmann_method#Criticism

http://www.securityfocus.com/brief/888

http://csrc.nist.gov/publications/nistpubs...800-88_rev1.pdf

Data Recovery Services:

http://www.oneworlddatarecovery.com/

https://services.seagate.com/laptop_data_recovery.aspx

Do I need multiple passes to ensure my data is unrecoverable?

The answer is, probably not. I'm not an expert on the subject, but I've done some research. I haven't found any evidence that it's possible to recover data that has been overwritten. In fact, I found more evidence suggesting that it's technologically infeasible or at the very least hugely impractical.

I see it all the time. People suggest doing multiple passes to ensure the data is gone. On the other hand, I never see anybody suggesting that this is unnecessary and you only need to do one pass. I guess it's just not common knowledge, which is why I decided to write this article.

Where did this rumor come from?

I think credit can be given to Peter Gutmann. You may have heard of the Gutmann method, which performs 35 passes over data to ensure it can never be recovered. Peter Gutmann wrote a paper in 1996, and based on his research, he devised this method for wiping data on hard drives.

Back in 1996, Peter Gutman, computer science professor at Auckland University in New Zealand, published a paper proposing how data could be recovered from hard disk or floppy disk sectors that had been overwritten. The idea behind this is based on the fact that the read/write heads are never precisely positioned over the same exact area twice and that by using electron-microscopes (Scanning Tunneling Microscopy) it would be possible to find a 'shadow' of the previously written sector.

Source

So why isn't it possible to recover overwritten data?

The hard drives mentioned in this 1996 paper are MFM and RLL drives, which were the first generation of hard drives used for personal computers (IBM called them the Winchester drives). The largest MFM and RLL drives made got up to about 130MB in size and were quickly replaced by IDE/ATA hard drives. At the time Professor Gutman's paper was published, the MFM/RLL hard disk technology was already 10 years old.

Source

So it might be possible to recover overwritten data... if your hard drive is over 20 years old. The bit density of modern hard drives is much greater. A 2TB drive can hold over 15,000 times more data than the 130MB drives his research was based on. There simply isn't enough space between each bit for there to be a 'shadow copy' of the old data.

There are other issues as well. A single sector can be written to several times over the life of the hard drive. So you have multiple layers of old data to sift through, without a practical way of knowing in which order each layer of data was written.

Why does the Gutmann method have 35 passes?

Some people may argue that the 35 passes the Gutmann method uses is evidence that multiple passes is necessary to ensure data is unrecoverable. Actually, the reason why this method has 35 passes is because it was designed for several encoding schemes, for several different drive types. If you knew the encoding scheme your drive used, then you wouldn't need all 35 passes. But the Gutmann method was designed as an all-in-one, for many different encoding types.

Source

Each of patterns 5-31 was designed with a specific magnetic media encoding scheme in mind, which each pattern targets. The drive is written to for all the passes even though the table below only shows the bit patterns for the passes that are specifically targeted at each encoding scheme.

Debunking other myths

Peter Gutmann is popular for suggesting that companies and government agencies have special equipment for recovering overwritten data. However, there doesn't seem to be any evidence to suggest this is true.

To debunk the myth that the government has secret technology which we simply don't know about, consider this: The government will hire 3rd party companies to recover data for them. If the government has this supposedly great technology, why would they entrust potentially sensitive or classified information to a 3rd party corporation?

Why does the government require drives to be overwritten using multiple passes? The best explanation is paranoia. Simply the idea that it could be possible, though highly improbable, was enough for the government to require that data be wiped using multiple passes. The government, and some corporations, may go as far as degaussing the drives, throwing them in a shredder, or even incinerating them.

I've read articles where the writer attempted to find a data recovery service which claimed to recover data that has been completely overwritten. Of course, they found absolutely nothing. I decided to try for myself. I looked at a couple data recovery services, listed at the beginning of this article, to see if any of them claim to recover overwritten data. It's important to pay attention to the wording used, as 'overwritten' can refer to the partition or file tables being overwritten, and not the file data itself.

Source

Shock or force sustained; Files deleted or recycled; Computer won't boot; Crashed Hard Drive; Physical Damage; Failed External Storage Device; Exposure to extreme temperatures; Variations in humidity; Degrading read/write heads; Moisture contact or penetration; Spike or surge in power source; Failed RAID Server; Hard Drive seen in the BIOS but not recognized; Broken or corrupted RAID array configuration; File System formatted; Operating System reinstalled; Virus intrusion Corrupt partition and boot tables; Software (ScanDisk, CheckDisk, Defrag); Defective mechanical or electronic components; Logical Corruption; Accidental Deletion; Accidental Formatting; Bad sectors; Clicking/scratching noises

Source

Dropped notebook - no longer turns on; Inaccessible drives and partitions; Applications that are unable to run or load data; Corrupted data; Virus attacks; Hard disk component failure; Hard disk crashes; Damage due to fire or liquids; Media surface contamination and damage; Accidental reformatting of partitions; Accidental deletion of data

None of these scenarios refers to recovering data that was explicitly overwritten. Rather, many of these scenarios refer to physical damage or failure of the drive. Other scenarios, such as reformatting of partitions, does not overwrite the data.

Is there any reason to do multiple passes?

Some people will suggest doing multiple passes to ensure the data is overwritten correctly. Technology can malfunction, which means data may not be overwritten correctly on the first pass. So hopefully it will be overwritten on the second or third pass.

There is an alternative. Rather than doing multiple passes, some software can verify the data was overwritten correctly, by reading the data after it's written. For me, this has two advantages:

1. It's faster than doing multiple passes. It will only go over the data twice, once to overwrite it, and once to check it was overwritten correctly.
   
2. It verifies all of the data was overwritten correctly. If the hard drive malfunctions once, it can do it again. So even after 10 passes, you're not guaranteed the data was overwritten correctly.
   

So if there is a small block of data that was not overwritten, the verify round will catch it, and it will overwrite the data as many times as necessary until it finally succeeds.

If the software you are using does not have a verify feature, and you don't feel safe doing just one pass, then maybe it's OK to perform multiple passes over the data to ensure everything is overwritten at least once. But the idea that there may be shadow / ghost copies of old data simply isn't true on modern, high-density hard drives.

Is there any reason to degauss / physically destroy a hard drive?

Actually, yes. When a hard drive gets a bad sector, the chip in the drive remaps the bad sector to a good sector. Then the bad sector is never touched again.

Because of this, the bad sector is never written to, making it impossible to overwrite, no matter how many passes you use. Data recovery services can recover data from bad sectors, so the only way to eliminate them is to degauss or physically destroy the drive.

What about floppy disks / ZIP disks?

Yes, you should overwrite these disks several times. Floppy disks and ZIP disks are old technology. They're magnetic storage with a low bit density, so it could be very much possible to recover old data stored on one of these disks if it's not overwritten several times.

What about flash memory / SSDs?

I honestly have no idea. This article refers to overwriting data on hard drives. Flash memory is completely different technology, and I wasn't able to find any solid information related to this topic. If somebody could fill me in, I would be happy to add it to my article.

Update: This news story explains the difficulty of securely wiping data on flash drives. It doesn't explain anything about shadow copies. But flash drives have limited read/write cycles, so flash drives will constantly remap sectors to extend the life of the device. The result is that overwriting data on a flash drive or SSD may not actually wipe the data. My recommendation in this case is to use full drive encryption.

http://www.theregister.co.uk/2011/02/21/flash_drive_erasing_peril/

How do I configure Eraser to do one pass?

Eraser - http://eraser.heidi.ie/

To configure Eraser to do just one pass, in the File menu, click Edit >> Preferences >> Erasing.

There are two separate tabs, one for Files and another for Unused Disk Space. In both of these tabs, select "Pseudorandom Data", 1 pass.

How do I configure CCleaner to do one pass?

CCleaner - http://www.piriform.com/ccleaner

CCleaner is easier to configure.

Options >> Settings

Select "Secure File Deletion"

Choose "Simple Overwrite (1 pass)"

How do I configure Darik's Boot and Nuke (DBAN) to do one pass?

Darik's Boot and Nuke - http://www.dban.org/

On the first screen, simply press ENTER to pass this screen. DO NOT type AUTONUKE.

On the next screen, press M on your keyboard to choose a method. Then choose PRNG Stream and press SPACE.

Finally, press R to change the number of rounds. Ensure this value is set to 1 and press ENTER.

Conclusion

Several years ago, it may have been theoretically possible to recover data from a hard disk, when bit density was much lower and so there was more space between bits for old data to leave "residue" behind. Today, the bit density of hard drives is much greater, to the point where just one pass will effectively erase data, making it impossible to recover the data using even the most advanced technology.

However, even with multiple passes, it's no guarantee that all of the data will be overwritten correctly. Bad sectors are remapped on hard drives, leaving old data behind which is impossible to overwrite. The only way to eliminate bad sectors is to degauss or physically destroy the drive.

So stop telling people to overwrite data 3, 7, 35 times. It's not necessary and a waste of time. And if a person is not easily convinced, refer them to this article, it's why I wrote it.

If you're interested in learning more, I suggest reading this article. It mentions several more points that I didn't cover in my article.

http://www.nber.org/sys-admin/overwritten-data-gutmann.html

Quotes

I don't make the best author on the subject. So instead, here are some of my favorite quotes from the sources listed at the beginning of this article, written by people who are more qualified than I am.

http://www.securityfocus.com/brief/888

In research published on Thursday, auditor Craig Wright tested the ability of a special type of electron microscope, known as a magnetic force microscope, to read data that has been erased. While overwriting the data multiple times with a random series of 0s and 1s makes it harder to recover, Wright found that it is nearly impossible to recover any meaningful amount of data after a only single pass. Recovering a single byte of data, for example, on a used drive is successful less than one percent of the time, he found. Accurately recovering four bytes, or 32 bits, of data only works nine times out of each million tries.

"In many instances, using a MFM (magnetic force microscope) to determine the prior value written to the hard drive was less successful than a simple coin toss."

http://en.wikipedia.org/wiki/Gutmann_method#Criticism

In the time since this paper was published, some people have treated the 35-pass overwrite technique described in it more as a kind of voodoo incantation to banish evil spirits than the result of a technical analysis of drive encoding techniques. As a result, they advocate applying the voodoo to PRML and EPRML drives even though it will have no more effect than a simple scrubbing with random data. In fact performing the full 35-pass overwrite is pointless for any drive since it targets a blend of scenarios involving all types of (normally-used) encoding technology, which covers everything back to 30+-year-old MFM methods (if you don't understand that statement, re-read the paper). If you're using a drive which uses encoding technology X, you only need to perform the passes specific to X, and you never need to perform all 35 passes. For any modern PRML/EPRML drive, a few passes of random scrubbing is the best you can do. As the paper says, "A good scrubbing with random data will do about as well as can be expected". This was true in 1996, and is still true now.

http://www.nber.org/sys-admin/overwritten-data-gutmann.html

Gutmann claims that "Intelligence organisations have a lot of expertise in recovering these palimpsestuous images." but there is no reference for that statement. There are 18 references in the paper, but none of the ones I was able to locate even referred to that possibility. Subsequent articles by diverse authors do make that claim, but only cite Gutmann, so they do not constitute additional evidence for his claim.

...

In fact, data written to the disk prior to the data whose recovery is sought will interfere with recovery just as must as data written after - the STM microscope can't tell the order in which which magnetic moments are created. It isn't like ink, where later applications are physically on top of earlier markings.

Added two more sources, thanks to BudMan

Added 'Why does the Gutmann method have 35 passes?'

Added 'Is there any reason to do multiple passes?'

Added 'Is there any reason to degauss / physically destroy a hard drive?'

Added 'What about flash memory / SSDs?'

Updated conclusion

Added Quotes

Added 'What about floppy disks / ZIP disks?'

Fixed images

Edited February 22, 2011 by Xinok

[kimsland]

Excellent guide, I'll probably end up referring to in future (Y)

But, the reason for more than 1 pass (as stated many times across the net) is because sometimes a sector (or bit or whatever) is missed in the wipe

This is also apparent in the slack space of occupied clusters, as well.

Therefore multiple passes are suppose to eliminate this issue of "missed" areas

Plus it is guaranteed that multiple passes (maximum, and unbelievable, and not practically possible) 35 passes will confirm it is wiped (note 3 preferred)

Just as an added point to "wiping" Generally Users who wipe their hard drive are trying to stop any recovery of private data. But, the only secure method of doing this (without any possibility of recovery) is to physically destroy the Hard drive!

Since Hard Drives are basically cheap these days, I'd go with the physical destroy option if you are really paranoid

Anyway, Hard Drives have these super magnets inside that are great to have fun with :)

[+Xinok Subscriber²]

But, the reason for more than 1 pass (as stated many times across the net) is because sometimes a sector (or bit or whatever) is missed in the wipe

This is also apparent in the slack space of occupied clusters, as well.

Therefore multiple passes are suppose to eliminate this issue of "missed" areas

Plus it is guaranteed that multiple passes (maximum, and unbelievable, and not practically possible) 35 passes will confirm it is wiped (note 3 preferred)

I added this to my article. Though I concentrated more on the fact that some software (including DBAN) can verify the data is overwritten correctly, making multiple passes unnecessary for this reason.

[LittleNeutrino Veteran]

the only true way to destroy your data is to destroy the physical media that it is one. and only then it makes it difficult however it might still be possible to recover it. depending on how it was destroyed and what the type of physical media was.

[hdood]

But, the reason for more than 1 pass (as stated many times across the net) is because sometimes a sector (or bit or whatever) is missed in the wipe

This is also apparent in the slack space of occupied clusters, as well.

What do you base this claim on? Why would the drive miss a write? Are you claiming that a hard drive randomly misses writes? That sounds like it would be wonderful for data integrity. The "slack space of occupied clusters" sounds like something that would be file system related and completely meaningless when it comes to doing raw writes to the disk (ie zeroing an entire partition or drive).

Maybe I'm just misunderstanding you and you are talking exclusively about tools that use normal file system operations to write to the drive in an attempt to overwrite files on an existing file system? Even then it's a dubious claim.

the only true way to destroy your data is to destroy the physical media that it is one. and only then it makes it difficult however it might still be possible to recover it. depending on how it was destroyed and what the type of physical media was.

Physically destroying the drive only makes it "difficult" to recover data? You're way off scale. Overwriting the data a single time makes it impossible to recover it with any known technique. Multiple passes makes it likely impossible even with any possible future techniques. Physically destroying the drive leaves nothing behind.

[Hell-In-A-Handbasket]

imho best thing is to physically destroy the platter.

Edited November 18, 2009 by Hell-In-A-Handbasket

[LightIdea]

Uh oh the van is near.

[Raa]

Considering you opened that article with "Probably not ... I'm no expert" i'm going to have to take your report with a grain of salt.

Good read though, thanks.

[+Gary7 Subscriber²]

A sledge hammer works wonders with destroying data on a HD.

[mokthraka]

A sledge hammer works wonders with destroying data on a HD.

then put the remains in a mircrowave for 1 minute or 2. NOTHING is going to be recoverable then.

or just open the hard drive take the disks out and put them in the microwave. either way works

If you want to make sure there isnt anyway. just use a sander on it. cant recover anything from dust!

eeer to turn it to dust use an angle grinder

Edited November 19, 2009 by mokthraka

[Osiris]

I accidentally formatted and isntalled windows over the wrong drive a while back. I thought a windows format what a joke, should be able to restore the drive to pre-format state and get my data back no problems. Nope none of the data recovery places would touch it, best they did was give me a list of a cpl programs to try to recover stuff from the unused space. I tried a tonne of programs, In the end I managed to get a filetable and thought id done it, but nope I recovered a few in the non-overwritten space but the majority was lost to obvlivion.

I guess the point of the story being I concurr with the OPs research and my own that once its overwritten its pretty well gone, but if youre securely destroying data obviously a reason behind it so chances are youll continue to use multiple passes / encoding streams anyway.

....cant recover anything from dust!

What if they can?! Make sure you do your anglegrinding in different places guys :p

Edited November 19, 2009 by Osiris

[kimsland]

OK, so now we have the correct process:

Single Wipe of Hard Drive

Sledge Hammer until Hard Drive broken down to bits and pieces

You can Microwave at this point, but it may blow up your Microwave as well

Anglegrind, Note: Special emphasis to the platters. Until dust

Gather up all dust and throw out to sea (select a remote location 100Mile from land)

Let them try to recover that :p

[mokthraka]

I accidentally formatted and isntalled windows over the wrong drive a while back. I thought a windows format what a joke, should be able to restore the drive to pre-format state and get my data back no problems. Nope none of the data recovery places would touch it, best they did was give me a list of a cpl programs to try to recover stuff from the unused space. I tried a tonne of programs, In the end I managed to get a filetable and thought id done it, but nope I recovered a few in the non-overwritten space but the majority was lost to obvlivion.

I guess the point of the story being I concurr with the OPs research and my own that once its overwritten its pretty well gone, but if youre securely destroying data obviously a reason behind it so chances are youll continue to use multiple passes / encoding streams anyway.

What if they can?! Make sure you do your anglegrinding in different places guys :p

just cause you cant recover it does not mean someone else cannot.

but yeah make sure you grind in diffrent places. like mount Everest and some volcanoes

[DDStriker]

What do you base this claim on? Why would the drive miss a write? Are you claiming that a hard drive randomly misses writes? That sounds like it would be wonderful for data integrity. The "slack space of occupied clusters" sounds like something that would be file system related and completely meaningless when it comes to doing raw writes to the disk (ie zeroing an entire partition or drive).

Maybe I'm just misunderstanding you and you are talking exclusively about tools that use normal file system operations to write to the drive in an attempt to overwrite files on an existing file system? Even then it's a dubious claim.

Physically destroying the drive only makes it "difficult" to recover data? You're way off scale. Overwriting the data a single time makes it impossible to recover it with any known technique. Multiple passes makes it likely impossible even with any possible future techniques. Physically destroying the drive leaves nothing behind.

+1

"missed a sector" made me laugh

also as for a "bad sector" wouldn't that be triggered due to the fact the hard drive could not write to it at all? meaning it updates the index to say "don't use this sector its bad use another one"

so even if they could read the bad sector it should be blank? I'm not 100% on that but seems logical

imho best thing is to physically destroy the platter.

Not really i've seen them piece together a floppy disk that was cut up into tiny random sized pieces and read the data from it (discover channel)

a simple wipe to the disk would be more then sufficient otherwise i'd see manufactures trying to take advantage of this (multi layered writes :laugh:)

writing a bunch of 0's is more effective then just destroying the disk and a combination of both is overkill

unless you are worried about them finding out how long your drive has been running for :D something to be concerned about if you're trying to make it seem like a drive never existed

[kimsland]

+1

"missed a sector" made me laugh

Try this on a Hard drive you don't care about much

Start the Wipe and then physically start shaking the Hard Drive (really hard, for about 10 mins)

Then run the Verifier, it will likely say some areas were not successfully wiped

So you say, What "shake" the drive?

Yes just like how some very old (or well used) drives tend to clatter and chunk away at times, sometimes a wipe area is missed

And this is why there are multiple wipes stated to do, by everyone, including even in the settings of the programs themselves!

Why the hec have the makers of these wipe programs made extra wipe options? You really must wonder.

also as for a "bad sector" wouldn't that be triggered due to the fact the hard drive could not write to it at all? meaning it updates the index to say "don't use this sector its bad use another one"

so even if they could read the bad sector it should be blank? I'm not 100% on that but seems logical

Unless of course that Bad spot was not bad to begin with, and data was originally written to it (highly likely) And therefore the bad spots still count

unless you are worried about them finding out how long your drive has been running for :D something to be concerned about if you're trying to make it seem like a drive never existed

Yes this is important too. Legally the drive identifier (Mac address) must line up with what was seen to be used (by online servers), whilst a User browsed i11egal sites. Plus also other hardware like your Motherboard and network card.

So I hope that may help your "+1" to be a little bit more understood ;)

[Azusa]

[hdood]

Try this on a Hard drive you don't care about much

Start the Wipe and then physically start shaking the Hard Drive (really hard, for about 10 mins)

Are you kidding? Shake the drive really hard for ten minutes? That would likely physically damage it permanently. No **** it would have a hard time accessing it.

Yes just like how some very old (or well used) drives tend to clatter and chunk away at times, sometimes a wipe area is missed

I don't see how you can claim that that follows. A drive making certain noises does not mean it is missing writes. By that logic regular writes of files would also randomly fail, something that would be a rather serious integrity problem.

[kimsland]

By that logic regular writes of files would also randomly fail, something that would be a rather serious integrity problem.

Yes correct :)

Thank god the program manufacturer put extra multiples of wipes in their program, just for this single reason - missed areas on the drive

Glad that's resolved ;)

[XerXis]

Are you kidding? Shake the drive really hard for ten minutes? That would likely physically damage it permanently. No **** it would have a hard time accessing it.

I don't see how you can claim that that follows. A drive making certain noises does not mean it is missing writes. By that logic regular writes of files would also randomly fail, something that would be a rather serious integrity problem.

which would ofcourse be one reason to throw it away after securely wiping it. That's the whole point, you have a drive gone bad and want to throw it away after being sure nobody can retrieve data from it.

[+Xinok Subscriber²]

http://en.wikipedia.org/wiki/Error_correction_codes

Error Correction Codes (ECC) are algorithms which allow you to repair data after it has been corrupted. Everything from hard drives, to CDs and DVDs, to flash memory, to DTV (Digital TV) and HDRadio, to satellite communications, to GSM (cell phone) and everything inbetween use ECC to recover from errors.

So yes, hard drives mess up writes. Then it uses ECC to repair and recover the original data, which is why you don't just lose your data.

Your HDD SMART info has an ECC attribute, if you care to take a look.