Problem:
HP DL/585 with large disks (ie 4x300GB), using the hardware RAID. When performing large reads or writes (ie performing database backups, trying to block-write a NetVault disk device, etc) the load on the computer becomes very high; wait-on-io times jump to 60-80% of CPU usage; and out-of-band reads or writes (ie doing an ls, or opening a file for edit) become extremely slow.
One opinion
...from Red Hat Enterprise Linux 4 (Nahant) Discussion List:
Well, having extremely high IO wait during heavy writes is technically
"normal". Based on what you are doing, copy data from an extremely fast
disk subsystem, to local disk (which are likely much slower) you are
going to saturate the IO bandwidth of the local disk and thus your
system will spend a lot of time waiting for writes to complete.If you are saturating the disks with writes, then it's likely that reads
will be very slow because they queue up behind all the writes. This can
be especially true if you local disk are connected to a RAID controller
with a largish cache (say 256MB, which seems common now adays). Linux
will keep the 256MB of cache full with writes, and when it needs to read
even a small amount of data the request will be queued behind all of
that data.