Status and plans. Tier1 Storage. on behalf of. June '08

Transcript

1 Instituto Nazionale di Fisica Nucleare Centro Nazionale per la Ricerca e Sviluppo nelle Tecnologie Informatiche e Telematiche CASTOR2@CNAF CASTOR@CNAF Status and plans Dejan Vitlacil on behalf of Tier1 Storage June '08

2 Overview Storage & CNAF Setup History Future SRM Top 5

3 Storage classes Storage & CNAF Implementation of 3 (2) Storage Classes needed for LHC Disk0Tape1 (D0T1) CASTOR Disk1Tape0 (D1T0) GPFS/StoRM Disk1Tape1 (D1T1) GPFS/StoRM

4 CASTOR2 Setup Production Instance 4+1 core servers (stager, lsf, dlf, name-server) all SLC3 32-bit - 1 SLC4 tmp Name server, Stager Oracle RAC (SRMv.2.2 DB & Repack, Single Instance) 1 server for ACSLS (library management) 1 library: STK L5500 (1 PB on-line) 15 tape servers ( b di drives, 6 LTO2 di drives) 6 servers srm v servers srm v disk-servers servers all XFS (nightly defragmentation) SLC4 32and 64-bit 2 xrood servers (peer and manager) Tools, Monitoring and Alarms QUATTOR configuration and installation LEMON for fabric monitoring (want to try LAS) House made monitoring tools for some Castor parts NAGIOS for proactive alarms notification (SMS in progress) Exclusion of problematic servers from service (via dns alias)

5 CASTOR Setup (1) Sun Blade v100 with 2 internal ide disks with software raid-1 running ACSLS 7.0 OS Solaris 9.0 Setup Core services are on machines with scsi disks, hardware raid1, redundant power supplies tape servers and disk servers have lower level hardware, like WNs 40 disk servers attached to a SAN full redundancy FC 2Gb/s or 4Gb/s (latest...) connections (dual controller HW and Qlogic SANsurfer Path Failover SW or Vendor Specific Software) STK L5500 silos (5500 slots, partitioned wth2formfactor slots, about 2000 LTO2 for and B, 200GB cartridges, tot capacity ~1.1PB non compressed ) 6 LTO B drives, 2 Gbit/s FC interface, MB/s rate. SAN 15 tape servers STK FlexLine 600, IBM FastT900, EMC Clarion

6 CASTOR Setup (2) Setup CASTOR core services v on 5 machines castor-6: rhserver, stager, rtcpclientd, cleaningdaemon, MigHunter, RecHunter, expert castorlsf01: Master LSF, rmmaster dlf01: dlfserver, Cmonit, Oracle for DLF castor-8: nsdaemon, vmgr, vdqm, msgd, cupvd castor-4: mighunter SLC4 SRM v2.2. Oracle DB (Single Instance) Name server Oracle DB (Oracle RAC) Stager Oracle DB (Oracle RAC) 1 SRMv1 endpoints, DNS balanced: srm://castorsrm.cr.cnaf.infn.it:8443 (used for disk pools with tape backend) HOPE TO BE DISSMISSED SOON! 2 SRMv22 endpoint: srm://srm-v2.cr.cnaf.infn.it:8443 cnaf it:8443 srm://srm-cms-v2.cr.cnaf.infn.it:8443 R ko l DB Repack Oracle DB (Single Instance)

7 Castor setup (3) Setup svcclass exp diskpool diskservers size (TB) alice ALICE alice cms CMS cms atlas ATLAS atlas atlas_pin ATLAS atlas lhcb LHCb lhcb lhcb_raw LHCb lhcb_raw argo ARGO argo ams AMS ams1 1 3 pamela PAMELA pamela1 1 4 magic MAGIC babar BABAR lvd LVD archive virgo VIRGO cdf CDF Probably will change as soon as new storage arrive 1.6 PB ( not all for Castor )

8 CASTOR blues History Upgrade from v to v (July 9-11 '07) New LSF plugin load on Oracle db lowered Previously frequent rmmaster meltdowns with # PEND jobs > 1000 Increased stability of scheduler (new job limit 30000) Good performances Upgrade to v (September '07) D1T0 pools management still problematic Not properly managed disks with no space left (requests still queued) Nearly impossible for the user to delete files from disk Situation between Stager, NameServer and Diskservers is not synchronized tape recall not yet enough efficient i Still needs for visible workarounds Manual load distribution on disk servers. Not negligible administrative effort for db clean-up Repack not yet supported

9 CASTOR blues (2) History Upgrade to v (November '07) Hotfix requested after installation due to FTS (dcache SRM) Still problems with Repack but with dev. team support, it was usable BugCleaning script + other workaround scripts used heavily Upgrade to v (May '07) Much more stable instance Much more usable Repack SLC4 was necessary for migration and streaming policy Great step -> integration of workarounds ( now is turn of Op. Team scripts :-) ) Great appreciation for your work CCRC went well and Castor did his part very well at CNAF Still some experiment specific tuning is required (perfection needed)

10 Our next Castor steps Future Get rid of SRM v1.1 as soon as possible Move Core Services on new SLC4 machines New library (SUN is on site from ) 8 tape drives (T10000; 500GB) 8 tape servers (probably bl we'll try 64-bit SLC4) 7000 slots (expandable to ) Oher 12 drives (1TB) in november Try to integrate Castor Lemon Sensor Try Lemon LAS Develop some more tools to keep Castor under control New Storage 1.6 PB - should arrive soon (end of July)

11 SRM SRM at CNAF We'd like to quit using srmv1.1 by the end of June (very difficult but likely) We are still running Some problems: DB deadlocks BE crashes Great help from Shaun GodBlessHim DeWitt We'd like to implement a solution with multiple BE as soon as possible

12 Top 5 ( or what bother us) Top 5 Stream and Migration Control total control we want to pilot and control everything GC Policy It seems like a step backwards (sysadmin lose control) Very expensive for the system Castor monitoring tool (web-based) Maybe with common effort Lemon sensor for Castor is CC Castor Workarounds Control We W all need to run the same Castor Clean the not necessary nsenterclass, svcclass,..

13

14 Castor - ieri oggi e domani Castor è stato aggiornato alla versione (primi di aprile) Impostato ed installato il nuovo repack che ha mostrato maggiore stabilità della versione precedente ma ancora genera errori e richiede baby-sitting (metà di aprile) Afrontato il tema delle Migration policies e durante i test scoperto il baco che ne limita l'utilizzo solamente al SLC4 (i nostri core services sono con SLC3); per superarlo, installata una macchina a a parte SLC4 e spostati i servizi neccesari da far funzionare Migration Policies. (fine aprile e primi di maggio) Effetuata la prima configurazione delle Mig. Poli. Con CMS; create 8 tapepools (tape families); la posibilità di configurazione è molto limitata e richiede intervento lato amministratore; problema sollevato anche dal RAL; (primi di maggio e oggi) Dopo CCRC08 si farà l'upgrade alla versione <ultima-stabile> e tutte le macchine di core services verranno spostate su SLC4;