Amanda backup system is being revived, Re: [Triumf-linux-managers] Outage of the amanda backup system

[Konstantin Olchanski]

On Mon, Apr 23, 2007 at 08:45:22PM -0700, Konstantin Olchanski wrote:
> The amanda backup system is presently down, the main 5 TB
> raid array is unaccessible and no backups are happening.
> ... the last tape backup has been done around 2 April 2007.

The amanda backup system has been revived using higher capacity
disks and it has now performed about 4 backup cycles. It may take
the system a few more days before full backups are done for
all clients. I will post another message when that happens. Then,
it will take the system another 10 days or so to reach a steady state.

Last week, we made a few unsuccessful attempts at restoring
the failed raid array - for reasons unknown, there was no trace
of a valid filesystem on any disks. I suspect a catastrophic
malfunction of the sata_mv driver overwrote parts of the
filesystem. This driver came with an updated kernel and I now
reverted back to the proprietary "mv_sata" (notice different,
but similar name) that was in use prior to the kernel update.

We decided not to try harder to restore the failed filesystem,
but wait for the arrival of the new higher capacity disks.

The new 750 GB disks arrived on Thursday (as scheduled) and
by Friday night the new 11 TByte raid5 XFS filesystem was
burned in. Backup cycles have been running every day since
then and today I re-enabled the email notifications.

This is the current status of amanda:

- SL4.4 with latest SL4 kernel
- proprietary mv_sata driver for the two 8-port sata-raid cards
- XFS drivers from the SL4.4 contrib area (thanks to Denice D.)
- 16 SATA 750 GB disks (was 16 SATA 400 GB disks)
- software raid5 across 16 disks, 200-300 Mbyte/sec data transfer speed
- 11 TByte XFS filesystem (was 5.4 TByte ext3)
- amanda 2.5.1p2 (was 2.4.x)

The capacity upgrade from 5.4 to 11 TBytes will permit us to once
again backup all of trshare.

-- 
Konstantin Olchanski
Data Acquisition Systems: The Bytes Must Flow!
Email: olchansk-at-triumf-dot-ca
Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada