Stato computing e Fondo comune BaBar '04 M. Morandin CSN1-3 febbraio 2004 1
Sommario La transizione al nuovo modello di calcolo Stato dei centri Tier A statistiche di utilizzo delle risorse Situazione Fondi Comuni dopo l' IFC di gennaio 2
The big challenge for BaBar computing: moving from CM1 to CM2 CM1 produced a large number of physics results However as the luminosity increases, there are new challenges and we want to build on our experience The CM2 effort was designed to: Address areas which were probably not scalable with increased luminosity (e.g. Analysis model) Contain the costs with increasing luminosity (and perhaps generate some savings, experience will tell over time) Increase flexibility for the use of distributed computing Make BaBar physicists more productive and improve BaBar physics results 3
Hierarchical Eventstore/Mini The first computing model had foreseen a hierarchical eventstore with multiple data levels. The target (and achieved) sizes for real data (for example) were: Micro (Aod/Tag) ~ few kb/event (5-8kB/event in Objy depending on whether overheads are included, ~2kB/event in original Kanga) Mini/Esd data ~ 20kB/event (not implemented until 2002) Rec data ~ 100kB/event (150kB/event in Objy) Raw data ~ 30kB/event (50kB/event in Objy, 30kB in original xtc) In practice the Raw and Rec data were not usable due to the large size, eventstore access problems and deficiencies in their design BaBar physicists were stuck with two choices: either the Micro or the (original) raw data in xtc format Driven initially in part by the needs of calibrations, a new and compact mini format was developed in 2001/2002 and stored in the Mini/Esd component of the eventstore 4
First implementation of the Mini (original raw ~ 25kB/event) 2kB/event Micro AOD BtaCandidates P4 Quals Micro truth MC matching Tag Tag bits Tag floats and ints No Connections, duplicated data 8kB/event Mini This new compact mini/esd format solved the needs of calibrations, but still left a fundamental gap between the micro being used by the average analysis user and the esd 5 Esd Reco objects Tracks Fits Hits Emc clusters digis... PID information XxxPidInfo MC matching Tru GTracks GVerticies For CM2 we decided to improve on the micro implementation
New CM2 data organization Aod Reco objects Tracks Fits Emc clusters digis... PID information XxxPidInfo MC matching Tag Tag bits Tag floats and ints Micro Mini Tru GTracks GVerticies Cnd BtaCandidates Reco-based composite Usr Generic data event-based Candidate-based Esd Track hits Single-crystal clusters Dirc rings Dirc hits Drop original aod and replace it with the subset of the mini/esd to which it corresponds, the esd then becomes the remainder 6
Advantages of mini and new micro Small size of micro maintained with respect to original Kanga and improves on that of Objy eventstore (which was still outside of bounds of original cost model) Avoids unnecessary data duplication between data formats Make available a more detailed data content for analysis, calibrations, detector studies and event display Preserves connections between data levels (i.e. Aod and esd) Overall size of full mini small enough to be usable at a reasonable cost (original model with Raw/Rec was not in practice usable at any reasonable cost) 7
Old analysis model 8
New Skim Production user defined data introducing ntuple functionality 9
CM2 Status Main goal: produce physics results for the summer, using Run1-Run4 data Steps: process Run4 raw data and produce new Root based event store OK, running in Padova since September '03 convert Run1-3 Objy ev. store to new format OK, done at SLAC convert SP5 Monte Carlo Objy collection In progess at Lyon/SLAC skim Run1-3 data, producing deep copies and pointer collections Not yet started, two months to complete: critical! reprocess/skim Run4 data w/ final release and calibr. constants Started in Padova start SP6 with new event store Under final tests (not targeted for the summer) 10
Further develpments Beginning to look at options for migrating away from Objectivity for the non-eventstore databases Recommendation from last two reviews Locked into certain compiler/platform combinations Can't just move to the gcc3/intel compiler Only by removing the use of other databases can we save significantly on the Objectivity Licensing costs Project launched to gradually replace Objy in non- EventStore databases (Cond. DB, Conf DB, etc.) INFN contributes with 1->2 computing professionals 11
Red Hat Enterprise Linux Red Hat no longer releasing Desktop version of Linux Only Enterprise, list price of $299/box Fedora project offered a replacement for enthusiasts October HEPiX meeting discussed options Fedora/Debian/SuSE/RHEL Felt that porting large installations to new distribution would be a significant enterprise (pun intended) Conclusion: Should come to an acceptable agreement with RH for RHEL 12
Red Hat Enterprise Linux (Cont.) DoE (almost) has agreement in place for FY04 SP nodes and BaBar users at Universities SLAC & INFN Tier-As Other Tier-As will have own agreement In process of setting up build and batch facilities at SLAC RHEL is supported for 5 years 2.5 years new hardware support 2.5 years with security fixes only 13
Outline Resources and Activities at Tier A Sites USA: SLAC France: IN2P3 (Lyon) UK: RAL Italy: INFN (Padova, Bologna, Roma) Germany: GridKa (Karlsruhe) Site Utilization in Comparison with MOUs CSC compilation of Tier A data in normalized (i.e. comparable) CPU units Conclusions 14
INFN: Bologna (CNAF) Center Italian Tier-A for Analysis Built out of a former parking garage; halls just finished in November Ex-Rome farm was still to be installed; tape silo already up Also LHC Tier-1, and center for CDF, Virgo and others Will serve BaBar as the INFN Analysis Center to complement Padova Production 15
INFN: Status Tier-A now split across three centers Padova: Processing, Reprocessing, Skimming Roma: Monte Carlo Production Bologna (CNAF): Analysis Center Upgrades Status 2003: Upgrades expected by June (''new machines'') were delivered only at end of November (see next slide for impact) 2004-2005: Plan is that CPU/disk resources will scale at least as the integrated luminosity (and the MoU) Most of the resources for 2005 should be available already by July 2004 16
INFN: Status by Site Padova (Reconstruction, Skimming) Reprocessed slightly more than promised until summer 2003 (> 60 fb-1) Since Run 4 only Babar site doing Event Reconstruction ''old'' resources slightly more than enough for Run4 at the moment, allowing up to ~600 pb-1/day 4 farms capable of 150 pb-1/day each Upgrades for 2003 dedicated to skimming (about to start) An addional 2 farms (capable of 220 pb-1/day each) 17
INFN: Status (cont) Roma (Monte Carlo) Running with high efficiency throughout Late delivery of 2003 machines: Impact on Production partially compensated by parasitic production on Padova farms, but still only achieved 24M/month during last 5 months (expected ~30M/month) Bologna (Analysis) Old Tier-B farm has been installed New machines being installed: Site will be available (to selected users at first) some time in January/February 2004 18
Gridka: Activities Kanga-based analysis since 2001 Mainly used by German BaBar community Sizeable relief in CPU demands for RAL No Objy-based analysis performed in Germany Monte Carlo Production since November 2002 Initially SP4, SP5 since June 2003 Utilization has much improved with SP5 BaBar GRID Development since April 2003 19
Comparison of Tier A Usage As requested at the last IFC meeting, the CSC has compiled usage statistics for all Tier A sites in normalized units Actual usage data (as available) is shown in comparison with the MOU numbers More detail is available in the Tier A Usage spreadsheet Disk Space Disk space is the actual space made available to BaBar Normalized CPU times CPU times are those utilized by BaBar production and/or users CPU units have been normalized to a common denominator 20
SLAC Usage Disk space Batch time + Dedicated CPU SLAC - Disk Space SLAC - Batch Wall CLock Time + Dedicated CPU Actual MOU Actual MOU 300 250 200 150 100 50 0 10000 9000 8000 7000 6000 5000 1-Jan-2001 20-Jul-2001 5-Feb-2002 24-Aug-2002 12-Mar-2003 28-Sep-2003 15-Apr-2004 1-Nov-2004 1-Jan-2001 20-Jul-2001 5-Feb-2002 24-Aug-2002 12-Mar-2003 28-Sep-2003 15-Apr-2004 1-Nov-2004 TB 4000 3000 2000 1000 Normalized CPU Units 0 Week Starting Week Starting Extra disk space made available end of 2003 SLAC CPU time is mix of dedicated and batch use 21
Note: IN2P3 uses dynamic staging system (HPSS) Batch usage approaching MOU IN2P3 Usage Disk space Batch time IN2P3 - Disk Space IN2P3 - Batch Wall Clock Time Actual MOU Actual MOU 2500 2000 1500 1000 500 0 1-Jan-2001 20-Jul-2001 5-Feb-2002 24-Aug-2002 12-Mar-2003 28-Sep-2003 15-Apr-2004 1-Nov-2004 1-Jan-2001 20-Jul-2001 5-Feb-2002 24-Aug-2002 12-Mar-2003 28-Sep-2003 15-Apr-2004 1-Nov-2004 Normalized CPU Units 180 160 140 120 100 80 60 40 20 0 TB Week Starting Week Starting 22
Actual disk space exceeding 2003 MOU Batch time share near or above MOU RAL Usage Disk space Batch time UK - Disk Space UK - Batch Wall Clock Time Actual MOU Actual MOU 120 1600 100 80 60 40 20 1400 1200 1000 800 600 400 200 0 1-Jan-2001 20-Jul-2001 5-Feb-2002 24-Aug-2002 12-Mar-2003 28-Sep-2003 15-Apr-2004 1-Nov-2004 1-Jan-2001 20-Jul-2001 5-Feb-2002 24-Aug-2002 12-Mar-2003 28-Sep-2003 15-Apr-2004 1-Nov-2004 TB Normalized CPU Units 0 Week Starting Week Starting 23
No CPU purchase made in 2003 INFN Usage Disk space Dedicated CPU time INFN - Disk Space INFN - Dedicated CPU Actual MOU Actual MOU 2500 2000 1500 1000 500 0 1-Jan-2001 20-Jul-2001 5-Feb-2002 24-Aug-2002 12-Mar-2003 28-Sep-2003 15-Apr-2004 1-Nov-2004 1-Jan-2001 20-Jul-2001 5-Feb-2002 24-Aug-2002 12-Mar-2003 28-Sep-2003 15-Apr-2004 1-Nov-2004 Normalized CPU Units 90 80 70 60 50 40 30 20 10 0 TB Week Starting Week Starting Plan to reach 2005 level this year 24
Disk space reached MOU in 2003 Now reached MOU level with Skim Production GridKa Usage Disk space GridKa - Disk Space Actual MOU Batch time GridKa - Batch Wall Clock Time Actual MOU 80 1200 1000 800 600 400 200 0 1-Jan-2001 20-Jul-2001 5-Feb-2002 24-Aug-2002 12-Mar-2003 28-Sep-2003 15-Apr-2004 1-Nov-2004 1-Jan-2001 20-Jul-2001 5-Feb-2002 24-Aug-2002 12-Mar-2003 28-Sep-2003 15-Apr-2004 1-Nov-2004 Normalized CPU Units 70 60 50 40 30 20 10 TB 0 Week Starting Week Starting CPU usage took off in 2 nd half of 2003 with SP5 25
Tier-A status Tier A Sites All BaBar Tier A's appear to perform well High analysis load on RAL and IN2P3 INFN/Padova is reconstructing the new Run 4 data SLAC is converting 12-series data to CM2, IN2P3 joining GridKa is now skimming Distributed BaBar Today we calibrate our data at SLAC, reconstruct them in Padova, QA them back at SLAC, (re-)skim them in Karlsruhe+Lyon, analyze them at all five centers BaBar Computing has become more distributed than ever before! 26
27
Situazione Finanziaria Fondo Comune OCF 2004 in linea con bilancio preventivo di giugno '03 (2672 K$) inoltre richiesto contributo speciale per l' upgrade del Barrel IFR (piani di ottone): 121 K$ nel 2004 Credito IN2P3 proposto di considerare nel 2004 un credito derivante dal budget 2002, di circa 200 k$, derivante dal grande investimento in computing operato da IN2P3 in quell' anno; IFC aveva in passato rimandato una decisione finale sulla questione; proposta approvata di valutare i mezzi di calcolo corrispondenti a costi 2004 (48 k$) 28
Finanziamento ottone per upgrade Barrel IFR ottone per sestanti da rimpiazzare nel 2004 già acquisito da SLAC (121 K$) ottone per gli altri quattro sestanti sta per essere ordinato da noi (budget di 200 KEuro a Pisa) proposta INFN di essere esentati dal finanziamento dell' ottone non accettata da alcuni membri dell' IFC quota INFN ~ 50 KEuro INFN perciò contribuisce scalando dal credito che gli è riconosciuto per l' acquisto dei 2/3 dell' ottone 29
Share of brass for Barrel IFR upgrade C PhD Total Cost CY04 CY05 2004 2004(%) K$ (1/3) (2/3) Canada 14 3,30 11,93 3,97 7,96 China 5 1,20 4,34 1,44 2,90 France 40 9,40 33,98 11,29 22,68 Germany 17 4,00 14,46 4,81 9,65 Italy 75 17,70 63,98 21,27 42,71 The Netherlands 3 1,40 5,06 1,68 3,38 Norway 2 0,50 1,81 0,60 1,21 Russia 9 2,10 7,59 2,52 5,07 United Kingdom 40 9,40 33,98 11,29 22,68 United States 219 51,70 186,88 62,12 124,76 Total 424 100,00 364,00 121,00 243,00 SLAC INFN Overhead 9.8%TOTAL Brass 121,00 243,00 36,00 400,00 % 33,00 67,00 (2/3) 30
Computing Fund 2004 Computing Fund 2004 fornito ai referees e discusso prima della riunione dell' IFC budget globale rimasto quale era a giugno '03: ~ 5 M$ (-20% rispetto a 2003 con +100% luminosità acquisita nell' anno) significativa riduzione di investimento/luminosità quota SLAC per 30 fb -1 sensibilmente ridotta rispetto al '03 (1.8 M$ invece di 3.5 M$) sostituzioni macchine obsolete non automatico (solo vecchie box monoprocessore Sun nel 2004 a SLAC) per l' INFN: senza i Tier A europei: 549 k$ con la valutazione dei Tier A europei: 138 k$ 31
Riepilogo Situazione 2004 Fondo comune 2004 Totale INFN k$ k$ OCF 2672 473 + Brass 2793 473 + Computing Fund 7697 611 + IN2P3 credit (48 K$) 620 Situazione INFN k$ Credito da OCF '03 50 Anticipo 12/03 450 Da pagare 120 Risparmio cambio su anticipo 25 sblocco SJ a Padova (1.25 $/Euro) 80 32
CSC Bilancio computing non sufficientemente discusso nel Computing Steering Committee alcune modifiche importanti inserite all' ultimo momento alcuni punti controversi non sufficientemente approfonditi BaBar ha proposto di rinnovare il formato del Comitato per ottenere una più puntuale preparazione del budget per migliorare il coordinamento fra i Tier-A approvazione del nuovo CSC al prossimo IFC nuovo chair al posto di R. Mount per un anno (M.M.) Per giugno '04: bilancio preventivo computing '05 basato su CM2 occorrerà riformulare il contributo di SLAC (30 fb -1 ) si dovrà concordare l' estensione dell' impegno degli MoU agli anni 2006-2008. 33