Stato computing e Fondo comune BaBar '04

Transcript

1 Stato computing e Fondo comune BaBar '04 M. Morandin CSN1-3 febbraio

2 Sommario La transizione al nuovo modello di calcolo Stato dei centri Tier A statistiche di utilizzo delle risorse Situazione Fondi Comuni dopo l' IFC di gennaio 2

3 The big challenge for BaBar computing: moving from CM1 to CM2 CM1 produced a large number of physics results However as the luminosity increases, there are new challenges and we want to build on our experience The CM2 effort was designed to: Address areas which were probably not scalable with increased luminosity (e.g. Analysis model) Contain the costs with increasing luminosity (and perhaps generate some savings, experience will tell over time) Increase flexibility for the use of distributed computing Make BaBar physicists more productive and improve BaBar physics results 3

4 Hierarchical Eventstore/Mini The first computing model had foreseen a hierarchical eventstore with multiple data levels. The target (and achieved) sizes for real data (for example) were: Micro (Aod/Tag) ~ few kb/event (5-8kB/event in Objy depending on whether overheads are included, ~2kB/event in original Kanga) Mini/Esd data ~ 20kB/event (not implemented until 2002) Rec data ~ 100kB/event (150kB/event in Objy) Raw data ~ 30kB/event (50kB/event in Objy, 30kB in original xtc) In practice the Raw and Rec data were not usable due to the large size, eventstore access problems and deficiencies in their design BaBar physicists were stuck with two choices: either the Micro or the (original) raw data in xtc format Driven initially in part by the needs of calibrations, a new and compact mini format was developed in 2001/2002 and stored in the Mini/Esd component of the eventstore 4

5 First implementation of the Mini (original raw ~ 25kB/event) 2kB/event Micro AOD BtaCandidates P4 Quals Micro truth MC matching Tag Tag bits Tag floats and ints No Connections, duplicated data 8kB/event Mini This new compact mini/esd format solved the needs of calibrations, but still left a fundamental gap between the micro being used by the average analysis user and the esd 5 Esd Reco objects Tracks Fits Hits Emc clusters digis... PID information XxxPidInfo MC matching Tru GTracks GVerticies For CM2 we decided to improve on the micro implementation

6 New CM2 data organization Aod Reco objects Tracks Fits Emc clusters digis... PID information XxxPidInfo MC matching Tag Tag bits Tag floats and ints Micro Mini Tru GTracks GVerticies Cnd BtaCandidates Reco-based composite Usr Generic data event-based Candidate-based Esd Track hits Single-crystal clusters Dirc rings Dirc hits Drop original aod and replace it with the subset of the mini/esd to which it corresponds, the esd then becomes the remainder 6

7 Advantages of mini and new micro Small size of micro maintained with respect to original Kanga and improves on that of Objy eventstore (which was still outside of bounds of original cost model) Avoids unnecessary data duplication between data formats Make available a more detailed data content for analysis, calibrations, detector studies and event display Preserves connections between data levels (i.e. Aod and esd) Overall size of full mini small enough to be usable at a reasonable cost (original model with Raw/Rec was not in practice usable at any reasonable cost) 7

8 Old analysis model 8

9 New Skim Production user defined data introducing ntuple functionality 9

10 CM2 Status Main goal: produce physics results for the summer, using Run1-Run4 data Steps: process Run4 raw data and produce new Root based event store OK, running in Padova since September '03 convert Run1-3 Objy ev. store to new format OK, done at SLAC convert SP5 Monte Carlo Objy collection In progess at Lyon/SLAC skim Run1-3 data, producing deep copies and pointer collections Not yet started, two months to complete: critical! reprocess/skim Run4 data w/ final release and calibr. constants Started in Padova start SP6 with new event store Under final tests (not targeted for the summer) 10

11 Further develpments Beginning to look at options for migrating away from Objectivity for the non-eventstore databases Recommendation from last two reviews Locked into certain compiler/platform combinations Can't just move to the gcc3/intel compiler Only by removing the use of other databases can we save significantly on the Objectivity Licensing costs Project launched to gradually replace Objy in non- EventStore databases (Cond. DB, Conf DB, etc.) INFN contributes with 1->2 computing professionals 11

12 Red Hat Enterprise Linux Red Hat no longer releasing Desktop version of Linux Only Enterprise, list price of $299/box Fedora project offered a replacement for enthusiasts October HEPiX meeting discussed options Fedora/Debian/SuSE/RHEL Felt that porting large installations to new distribution would be a significant enterprise (pun intended) Conclusion: Should come to an acceptable agreement with RH for RHEL 12

13 Red Hat Enterprise Linux (Cont.) DoE (almost) has agreement in place for FY04 SP nodes and BaBar users at Universities SLAC & INFN Tier-As Other Tier-As will have own agreement In process of setting up build and batch facilities at SLAC RHEL is supported for 5 years 2.5 years new hardware support 2.5 years with security fixes only 13

14 Outline Resources and Activities at Tier A Sites USA: SLAC France: IN2P3 (Lyon) UK: RAL Italy: INFN (Padova, Bologna, Roma) Germany: GridKa (Karlsruhe) Site Utilization in Comparison with MOUs CSC compilation of Tier A data in normalized (i.e. comparable) CPU units Conclusions 14

15 INFN: Bologna (CNAF) Center Italian Tier-A for Analysis Built out of a former parking garage; halls just finished in November Ex-Rome farm was still to be installed; tape silo already up Also LHC Tier-1, and center for CDF, Virgo and others Will serve BaBar as the INFN Analysis Center to complement Padova Production 15

16 INFN: Status Tier-A now split across three centers Padova: Processing, Reprocessing, Skimming Roma: Monte Carlo Production Bologna (CNAF): Analysis Center Upgrades Status 2003: Upgrades expected by June (''new machines'') were delivered only at end of November (see next slide for impact) : Plan is that CPU/disk resources will scale at least as the integrated luminosity (and the MoU) Most of the resources for 2005 should be available already by July

17 INFN: Status by Site Padova (Reconstruction, Skimming) Reprocessed slightly more than promised until summer 2003 (> 60 fb-1) Since Run 4 only Babar site doing Event Reconstruction ''old'' resources slightly more than enough for Run4 at the moment, allowing up to ~600 pb-1/day 4 farms capable of 150 pb-1/day each Upgrades for 2003 dedicated to skimming (about to start) An addional 2 farms (capable of 220 pb-1/day each) 17

18 INFN: Status (cont) Roma (Monte Carlo) Running with high efficiency throughout Late delivery of 2003 machines: Impact on Production partially compensated by parasitic production on Padova farms, but still only achieved 24M/month during last 5 months (expected ~30M/month) Bologna (Analysis) Old Tier-B farm has been installed New machines being installed: Site will be available (to selected users at first) some time in January/February

19 Gridka: Activities Kanga-based analysis since 2001 Mainly used by German BaBar community Sizeable relief in CPU demands for RAL No Objy-based analysis performed in Germany Monte Carlo Production since November 2002 Initially SP4, SP5 since June 2003 Utilization has much improved with SP5 BaBar GRID Development since April

20 Comparison of Tier A Usage As requested at the last IFC meeting, the CSC has compiled usage statistics for all Tier A sites in normalized units Actual usage data (as available) is shown in comparison with the MOU numbers More detail is available in the Tier A Usage spreadsheet Disk Space Disk space is the actual space made available to BaBar Normalized CPU times CPU times are those utilized by BaBar production and/or users CPU units have been normalized to a common denominator 20

21 SLAC Usage Disk space Batch time + Dedicated CPU SLAC - Disk Space SLAC - Batch Wall CLock Time + Dedicated CPU Actual MOU Actual MOU Jan Jul Feb Aug Mar Sep Apr Nov Jan Jul Feb Aug Mar Sep Apr Nov-2004 TB Normalized CPU Units 0 Week Starting Week Starting Extra disk space made available end of 2003 SLAC CPU time is mix of dedicated and batch use 21

22 Note: IN2P3 uses dynamic staging system (HPSS) Batch usage approaching MOU IN2P3 Usage Disk space Batch time IN2P3 - Disk Space IN2P3 - Batch Wall Clock Time Actual MOU Actual MOU Jan Jul Feb Aug Mar Sep Apr Nov Jan Jul Feb Aug Mar Sep Apr Nov-2004 Normalized CPU Units TB Week Starting Week Starting 22

23 Actual disk space exceeding 2003 MOU Batch time share near or above MOU RAL Usage Disk space Batch time UK - Disk Space UK - Batch Wall Clock Time Actual MOU Actual MOU Jan Jul Feb Aug Mar Sep Apr Nov Jan Jul Feb Aug Mar Sep Apr Nov-2004 TB Normalized CPU Units 0 Week Starting Week Starting 23

24 No CPU purchase made in 2003 INFN Usage Disk space Dedicated CPU time INFN - Disk Space INFN - Dedicated CPU Actual MOU Actual MOU Jan Jul Feb Aug Mar Sep Apr Nov Jan Jul Feb Aug Mar Sep Apr Nov-2004 Normalized CPU Units TB Week Starting Week Starting Plan to reach 2005 level this year 24

25 Disk space reached MOU in 2003 Now reached MOU level with Skim Production GridKa Usage Disk space GridKa - Disk Space Actual MOU Batch time GridKa - Batch Wall Clock Time Actual MOU Jan Jul Feb Aug Mar Sep Apr Nov Jan Jul Feb Aug Mar Sep Apr Nov-2004 Normalized CPU Units TB 0 Week Starting Week Starting CPU usage took off in 2 nd half of 2003 with SP5 25

26 Tier-A status Tier A Sites All BaBar Tier A's appear to perform well High analysis load on RAL and IN2P3 INFN/Padova is reconstructing the new Run 4 data SLAC is converting 12-series data to CM2, IN2P3 joining GridKa is now skimming Distributed BaBar Today we calibrate our data at SLAC, reconstruct them in Padova, QA them back at SLAC, (re-)skim them in Karlsruhe+Lyon, analyze them at all five centers BaBar Computing has become more distributed than ever before! 26

27 27

28 Situazione Finanziaria Fondo Comune OCF 2004 in linea con bilancio preventivo di giugno '03 (2672 K$) inoltre richiesto contributo speciale per l' upgrade del Barrel IFR (piani di ottone): 121 K$ nel 2004 Credito IN2P3 proposto di considerare nel 2004 un credito derivante dal budget 2002, di circa 200 k$, derivante dal grande investimento in computing operato da IN2P3 in quell' anno; IFC aveva in passato rimandato una decisione finale sulla questione; proposta approvata di valutare i mezzi di calcolo corrispondenti a costi 2004 (48 k$) 28

29 Finanziamento ottone per upgrade Barrel IFR ottone per sestanti da rimpiazzare nel 2004 già acquisito da SLAC (121 K$) ottone per gli altri quattro sestanti sta per essere ordinato da noi (budget di 200 KEuro a Pisa) proposta INFN di essere esentati dal finanziamento dell' ottone non accettata da alcuni membri dell' IFC quota INFN ~ 50 KEuro INFN perciò contribuisce scalando dal credito che gli è riconosciuto per l' acquisto dei 2/3 dell' ottone 29

30 Share of brass for Barrel IFR upgrade C PhD Total Cost CY04 CY (%) K$ (1/3) (2/3) Canada 14 3,30 11,93 3,97 7,96 China 5 1,20 4,34 1,44 2,90 France 40 9,40 33,98 11,29 22,68 Germany 17 4,00 14,46 4,81 9,65 Italy 75 17,70 63,98 21,27 42,71 The Netherlands 3 1,40 5,06 1,68 3,38 Norway 2 0,50 1,81 0,60 1,21 Russia 9 2,10 7,59 2,52 5,07 United Kingdom 40 9,40 33,98 11,29 22,68 United States ,70 186,88 62,12 124,76 Total ,00 364,00 121,00 243,00 SLAC INFN Overhead 9.8%TOTAL Brass 121,00 243,00 36,00 400,00 % 33,00 67,00 (2/3) 30

31 Computing Fund 2004 Computing Fund 2004 fornito ai referees e discusso prima della riunione dell' IFC budget globale rimasto quale era a giugno '03: ~ 5 M$ (-20% rispetto a 2003 con +100% luminosità acquisita nell' anno) significativa riduzione di investimento/luminosità quota SLAC per 30 fb -1 sensibilmente ridotta rispetto al '03 (1.8 M$ invece di 3.5 M$) sostituzioni macchine obsolete non automatico (solo vecchie box monoprocessore Sun nel 2004 a SLAC) per l' INFN: senza i Tier A europei: 549 k$ con la valutazione dei Tier A europei: 138 k$ 31

32 Riepilogo Situazione 2004 Fondo comune 2004 Totale INFN k$ k$ OCF Brass Computing Fund IN2P3 credit (48 K$) 620 Situazione INFN k$ Credito da OCF '03 50 Anticipo 12/ Da pagare 120 Risparmio cambio su anticipo 25 sblocco SJ a Padova (1.25 $/Euro) 80 32

33 CSC Bilancio computing non sufficientemente discusso nel Computing Steering Committee alcune modifiche importanti inserite all' ultimo momento alcuni punti controversi non sufficientemente approfonditi BaBar ha proposto di rinnovare il formato del Comitato per ottenere una più puntuale preparazione del budget per migliorare il coordinamento fra i Tier-A approvazione del nuovo CSC al prossimo IFC nuovo chair al posto di R. Mount per un anno (M.M.) Per giugno '04: bilancio preventivo computing '05 basato su CM2 occorrerà riformulare il contributo di SLAC (30 fb -1 ) si dovrà concordare l' estensione dell' impegno degli MoU agli anni