Is usb 2.0 or disk itself bottleneck when writing data?

[lfm]

Hi

Having had an external disk for backups for some years.
When doing backups it writes rather fast at first - you get a rather short estimate for total time - but then after 5-10 minutes it slows to a halt almost.
Suspect internal cache is reason for this.

So actual writing to disk might be bottleneck - not usb 2.0.

So question arise whether disk itself is the problem?

Thinking to extend with USB 3.0 interface card and usb 3.0 compatible external disks.

Will backup go quicker then?

Not sure what is the way to go.

Thanks.

[chk071]

Hm, never had that with my external drives. My USB 2.0 drives always wrote with ~ 30 MByte/s, which is about what USB 2.0 can deliver. USB sticks are another thing though, the bottleneck is definitely the stick then.

BTW, if you use a program to do your backups, or a file sync software, definitely count that in as a potential culprit too. E.g. the program i use, FreeFileSync, is pretty slow when syncing the files.

[lfm]

Hm, never had that with my external drives. My USB 2.0 drives always wrote with ~ 30 MByte/s, which is about what USB 2.0 can deliver. USB sticks are another thing though, the bottleneck is definitely the stick then.

BTW, if you use a program to do your backups, or a file sync software, definitely count that in as a potential culprit too. E.g. the program i use, FreeFileSync, is pretty slow when syncing the files.

Thank you.
It's hdd based external drive. Rather cheap 2 TB at the time, so gather the drive itself is maybe 3200 rpm or similar.

I use Seagate DiskWizard which is free for those with Seagate disks.

Software matters for sure, remember my PS3 5 years ago changed with a firmware update. From usuall 2.5 hours it suddenly took 16-20 hours - so efficient reading and caching while writing was removed, or similar. They needed room for PS Vita support of various sorts.

Not sure how much a disk can digest in itself?

Curious about how estimate is so high in beginning - which probably is that it is writing to drives cache - then it must actually put things on disk surfaces and that is slower. It looks like this.

So wonder if investment in usb 3.0 stuff all the way actually speed things up?
Or I can save that money for some better use.

[BertKoor]

Curious about how estimate is so high in beginning

In my experience this has to do with file size.

• When writing large files, the bottleneck is the transfer of data itself over the cable and/or the writing of that data to disk.
• With a large number of small files, the bottleneck is in the housekeeping of what file is where on the disk.

So if your backup starts with some very large files, it estimates all of them can be done quite fast. Then when it encounters thousands small files, it halts and the estimated time is adjusted to reflect that.

This is why I usually first make a zip archive on the C drive, and transfer that to the portable drive. That's just a transfer of one large file which is faster than the transfer of the individual files.

[lfm]

Curious about how estimate is so high in beginning

In my experience this has to do with file size.

• When writing large files, the bottleneck is the transfer of data itself over the cable and/or the writing of that data to disk.
• With a large number of small files, the bottleneck is in the housekeeping of what file is where on the disk.

So if your backup starts with some very large files, it estimates all of them can be done quite fast. Then when it encounters thousands small files, it halts and the estimated time is adjusted to reflect that.

This is why I usually first make a zip archive on the C drive, and transfer that to the portable drive. That's just a transfer of one large file which is faster than the transfer of the individual files.

Maybe it is like you say, that estimate is so simple going by number of files of total done.

I hoped that what is says estimate GB to backup would be the rule and how much is done.
I use max compression stuff, so usually half size to actually transfer.

After about 20 minutes or so I usually get close to reality estimate of remaining time - often about 3-4 hours.

I stay off the usb 3.0 investment now, and will evaluate a bit more.

I did a test some years ago copying a large file between disks on computer and to this external - and external took about 5 times longer. And a boot in between not to get read cache improvement on it.

My computer is bought 2010, so internal SATA is 3 Gbps. Modern machine has 6 Gbps.

But still don't know what was bottleneck - usb 2 or disk itself?

Upgraded internal disks a bit now for photo stuff, so backups will be larger always. So wanted to check out options to make it quicker.

[BertKoor]

FWIW, my external USB3 disk (WD "Passport") is significantly faster on a USB3 port than on a USB2 port.

The theoretical maximum transfer speed of USB2 is about 60 megabyte per second (480 Mbit/sec.)

[jdnz]

actually the max rate on usb2.0 is more like 45mb/s when you actually factor in the overheads (same as you can't actually push 1gb/s on gigabit ethernet - well not with any actual protocol)

a 'typical' 5400rpm drive should be able to do sustained write at about 90mb/s - so a good reason to upgrade to a modern (cheap) usb3 external for your backups, as it'll markedly cut your times for backup (presuming you're using decent backup software for the job that writes monolithic chunks)

[chk071]

So wonder if investment in usb 3.0 stuff all the way actually speed things up?
Or I can save that money for some better use.

The difference for me is that my USB 2.0 drive gives me a maximum of 30-35 MByte/s, while my USB 3.0 drives give me about 115 MByte/s. The difference is definitely noticable (after all it's more than 3 times the speed), and i wouldn't want to do without a 3.0 drive now. Dunno if external SSD's are even faster, or if the bottleneck is the USB port.

[jdnz]

Dunno if external SSD's are even faster, or if the bottleneck is the USB port.

samsung t1 (external USB) can hit 450mb/s - so pretty much the same as a sata samsung (makes sense - usb3.0 is up to 5gbit/s - not far off the 6gbit/s of sata3.0)

[lfm]

Thanks guys.

I did a check yesterday - 450 GB in 6 hours - so about 20 MB/s is what drive can take.
When this drive become more like 2 TB this is 24 hours for this drive alone.

So might be good with a usb 3.0 upgrade and new modern drives as well - in time.

I activated my old Acronis True Image 2016 yesterday to be able and make some incremental backups in between too, so backup times are kept decent. Free version Seagate discwizard did not allow that.

Waiting another year or two maybe multi terabyte SSD come down in price.

[jdnz]

if you want a big slab of storage (>2tb) and higher thruput a cheap USB3 attached would be the RAID to go, striping can get you past the ~100mb/s barrier most single HDDs have ('til the price of really big SSDs comes down from the stratosphere)

[lfm]

if you want a big slab of storage (>2tb) and higher thruput a cheap USB3 attached would be the RAID to go, striping can get you past the ~100mb/s barrier most single HDDs have ('til the price of really big SSDs comes down from the stratosphere)

Thanks, that sounds like a brilliant solution - many discs that each can mirror parts and multiply how much can be stored each second really maximizing use of usb 3. Probably SATA on computer that become bottleneck.

Are there RAID in the same cabinet - or is there software that can create an array of many separate standard external disks?

So having 2-3 usb ports for disks and maybe even usb 2.0 is enough then - and software manage mirroring parts of each file.

Sounds like way to go. And if specialized for backup only this is easier than a regular file system and how that must work.

Can you maybe do RAID volumes on Windows as it is now?
This must be investigated!!!!

[pekbro]

Another thing to consider is an anti-virus program, if you're running one
it can significantly delay general disc reads and writes. Generally though,
moving large amounts of data between discs, USB or no is not gonna go
super fast. USB is definitely the worst case though, IME.

-Cheers

[jdnz]

Are there RAID in the same cabinet - or is there software that can create an array of many separate standard external disks?

So having 2-3 usb ports for disks and maybe even usb 2.0 is enough then - and software manage mirroring parts of each file.

whilst there's nothing stopping anyone implementing 'softRAID' on usb I've never seen it done (well on windows, I'm sure someone's given it a go on linux using the md softraid driver).

and to get full speed you'd still need usb3 - as with usb2 the ports all go back to a hub, and the total thruput on the entire hub is the limiting factor

in hardwae usb3 raid Drobo (and to a lesser extent sonnettech) are the big players (also big $ - nice kit though, we've got a few Drobo's at work). Mid-range would things like the LaCie 2big series (2 bay usb3 raid) and the WD MyBook Duo range. At the cheap end you've got raid capable usb3 enclosures from the likes of Orico and StarTech

[lfm]

whilst there's nothing stopping anyone implementing 'softRAID' on usb I've never seen it done (well on windows, I'm sure someone's given it a go on linux using the md softraid driver).

and to get full speed you'd still need usb3 - as with usb2 the ports all go back to a hub, and the total thruput on the entire hub is the limiting factor

in hardwae usb3 raid Drobo (and to a lesser extent sonnettech) are the big players (also big $ - nice kit though, we've got a few Drobo's at work). Mid-range would things like the LaCie 2big series (2 bay usb3 raid) and the WD MyBook Duo range. At the cheap end you've got raid capable usb3 enclosures from the likes of Orico and StarTech

Looked at a WD MyBook Duo with 16 TB and 2 usb 3 ports today, about $600 seems having headroom - and was at delivery configured for striped raid, just plugin. A bit tempting. Since backup is a backup you don't need failure redundancy like if just part of regular system. Short backup times and good solution also encourage actually doing backups, and one day you are greatful you did.

But also a bit curious on dynamic drives or volumes for windows if they will work. Could buy two 3 TB and get a way with $200. So gettting a PCI Express usb 3 card whatever solution I go for.

I also have a support ticket at Acronis if they allow multiple jobs run simultaneously or not on their True Image software. If you configure a backup to go from one drive to backup drive - and another job in parallell between two other drives - would also speed things up - at least up to what my old SATA2 with 3 Gbps allow reading.

Thanks guys for all your input - really gave a boost to a solution, now with many options too.