Stato del progetto LHC Computing Grid LCG. Commissione I CERN. Alberto Masoni 1. last update: 01/07/ :04

Transcript

1 Stato del progetto LHC Computing Grid Alberto Masoni Commissione I last update: 01/07/ :04 Alberto Masoni 1

2 LCG LHC Computing Grid Project Goal of the project To prepare, deploy and operate the computing environment for the experiments to analyse the data from the LHC detectors Phase development of common applications, libraries, frameworks, prototyping of the environment, operation of a pilot computing service Phase acquire, build and operate the LHC computing service Alberto Masoni 2

3 GRID TECHNOLOGY IL PROGETTO EGEE I PROGETTI DATAGRID E DATATAG SI SONO CONCLUSI CON UNA VALUTAZIONE POSITIVA DELLA EU EGEE: Enabling Grids for E science in Europe E PARTITO IL I APRILE 2004 FINANZIAMENTO: ~32 MEUR (~4 ALL INFN) Alberto Masoni 3

4 IL PROGETTO LCG E I PROGETTI EGEE E GRID3 (US) LCG deve poter utilizzare le risorse per LHC su scala mondiale Necessità di interoperabilità fra le grids (e.g. LCG GRID3 attualmente in corso) LCG e EGEE condividono la responsabilità e il management dei seguenti settori: Middleware: EGEE partirà con il middleware package di LCG e l evoluzione sarà comune Infrastructure operation: LCG costituirà il nucleo fondamentale per lo sviluppo della grid EGEE Deployment of HEP application: integrazione con gli esperimenti LHC (EGEE NA4) Security: in fase di integrazione Alberto Masoni 4

5 EGEE ALL LCG LCG 2 & glite ACTIVITIES MEETING glite PROTOTIPO EGEE MIDDLEWARE (DISPONIBILE) Focus on analysis; strongly influenced by the ARDA RTAG Starts with components from AliEn, EDG, VDT and other projects Aim at addressing advanced requirements in particular from BioMedicals Prototyping short development cycles for fast user feedback Aim at delivering components compatible with LCG 2 wherever appropriate SA1 Preproduction Service Starts with LCG 2 Code base INFRASTRUTTURA DI PRODUZIONE EGEE Home for new development/reengineering required from LHC data challenges experiences Validation in particular by ATLAS/CMS current analysis tools Certification of promising selected components from glite Provided they satisfy Operations requirements LCG 2 INFRASTRUTTURA DI Current base for production services PRODUZIONE LCG Evolved with certified new or improved services from the preproduction Alberto Masoni 5

6 LCG 2 and Next Generation Middleware (LCG GDB ) LCG 2 focus on production, large scale data handling LCG-2 (=EGEE-0) 2004 prototyping prototyping EGEE ALL ACTIVITIES MEETING Next generation middleware focus on analysis The service for the 2004 data challenges Provides experience on operating and managing a global grid service Strong development programme driven by data challenge experience Evolves to LCG 3 as components progressively replaced with new middleware 2005 product product LCG-3 (=EGEE-x?) Alberto Masoni 6 Developed by EGEE project in collaboration with VDT (US) LHC applications and users closely involved in prototyping & development (ARDA project) Short development cycles Completed components integrated in LCG 2

7 Alcune considerazioni Portato avanti correttamente questo approccio consente: Di garantire il servizio per i Data Challenges degli esperimenti Di utilizzare in modo opportuno sia il software sviluppato fino ad ora sia i progressi legati allo sviluppo prototipale (ARDA) E necessario però un notevole sforzo di coordinamento e un grande impegno delle persone coinvolte nello sviluppo Le due linee non devono diventare convergenze parallele Software is not a technical problem. It is mainly a sociological problem. (Renè Brun seminario LCG Nov. 2001) Alberto Masoni 7

8 ARDA Long they laboured in the regions of Eä, which are vast beyond the thought of Elves and Men, until in the time appointed was made Arda... - J.R.R Tolkien, Valaquenta Alberto Masoni 8

9 ARDA in a nutshell Main activity: enable LHC analysis on the grid Use the grid software as it matures (EGEE project) ARDA should be the key player in the evolution from LCG2 to the EGEE infrastructure Provide early and continuous feedback (guarantee the software is what experiments expect/need) Use the last years experience/components both from Grid projects (LCG, VDT, EDG) and experiments middleware/tools (Alien, Dirac, GAE, Octopus, Ganga, Dial, ) Help in adapting/interfacing (direct help within the experiments) Every experiment has different implementations of the standard services, but: ARDA Used mainly in production environments Few expert users Coordinated update and read actions Interface with the EGEE middleware Verify (help to evolve to) such components to analysis environments Many users (Robustness might be an issue) Concurrent read actions (Performance will be more and more an issue) One prototype per experiment A Common Application Layer might emerge in future ARDA emphasis is to enable each of the experiment to do its job Provide a forum for discussion Comparison on results/experience/ideas Interaction with other projects Alberto Masoni 9 The experiment interfaces agree Experiment with the ARDA interfaces project leader the work plan and Piergiorgio Cerello (ALICE) coordinate David the Adams activity (ATLAS) on the experiment Lucia Silvestris side (users) (CMS) Ulrik Egede (LHCb)

10 LHCb The LHCb system within ARDA uses GANGA as principal component. The LHCb/GANGA plans: enable physicists (via GANGA) to analyse the data being produced during 2004 for their studies It naturally matches the ARDA mandate Have the prototype where the LHCb data will be the key (, RAL, ) At the beginning, the emphasis will be to validate the tool focusing on usability, validation of the splitting and merging functionality for users jobs Alberto Masoni 10 Grid activity: Use of the Glite testbed (since May 18 th ) Test jobs from Ganga to Glite Other contributions: GANGA interface to Condor (Job submission) and Condor DAGMAN for splitting/merging and error recovery GANGA Release management and software process LHCb Metadata catalogue tests Performance tests Collaborators in Taiwan (ARDA + local DB knowhow on Oracle)

11 CMS The CMS system within ARDA is still under discussion Provide easy access (and possibly sharing) of data for the CMS users is a key issue (Data management): RefDB is the bookkeeping engine to plan and steer the production across different phases (simulation, reconstruction, to some degree into the analysis phase). This service is under test It contained all necessary information except file physical location (RLS) and info related to the transfer management system (TMDB) The actual mechanism to provide these data to analysis users is under discussion Measuring performances underway (similar philosophy as for the LHCb Metadata catalog measurements) Exploratory/preparatory activity ORCA job submission to Glite Glite file catalog Reconstruction instructions RefDB McRunjob Tapes Reconstruction jobs Reconstructed data T0 worker nodes GDB castor pool Export Buffers RefDB in CMS DC04 Summaries of successful jobs Reconstructed data Checks what has arrived Updates RLS Transfer agent TMDB Updates Alberto Masoni 11

12 ATLAS The ATLAS system within ARDA has been agreed ATLAS has a complex strategy for distributed analysis, addressing different area with specific projects ( Starting point is the DIAL analysis model system (high level web services) The AMI metadata catalog is a key component Robustness and performance tests from ARDA In the start up phase, ARDA provided help in developing ATLAS production tools Being finalised Submission to Glite (test jobs OK, Athena jobs still not possible) First skeleton of high level services AMI Tests Alberto Masoni 12

13 ALICE Strategy: The ALICE/ARDA will evolve the ALICE analysis system (SuperComputing 2003) Where to improve: Strong requests on networking (inbound connectivity) Heavily connected with the middleware services Inflexible configuration No chance to use PROOF on federated grids like LCG in AliEn User libraries distribution Activity on PROOF Robustness and Error recovery Grid activity: C++ access library on Glite PROOF SLAVES ARDA ALREADY IN PHASE-3 OF THE PRESENT DATA CHALLENGE PROOF TcpRouter USER SESSION Site A PROOF MASTER SERVER Site B PROOF SLAVES PROOF SLAVES Site C Alberto Masoni 13

14 Slide Slide presentata presentata alla allacommisione CommisioneI I 24/9/03 24/9/03 DEPLOYMENT AREA: The LHC Grid Service Qui si parrà la tua nobilitate OBIETTIVIVO PRINCIPALE: FORNIRE UN SERVIZIO DI PRODUZIONE PER I DATA CHALLENGES NEL 2004 IL PLANNING ORIGINARIO PREVEDEVA: M 1.1 Luglio 2003 partenza del servizio pilota M 1.2 Fine 2003 disponibilità del sistema di produzione La prima milestone ha due mesi di ritardo, essenziallmente legati al ritardo di una componente di grid middleware (RGMA) la componente e stata sostituita con un altra, sia pure con minori funzionalita (MDS) grazie al fondamentale contributo INFN Cfr discussione ultima Commissione I La componente di grid middleware continua a mancare e non e chiaro se sara disponibile per la release da utilizzare nei Data Challenges degli esperimenti OBIETTIVO: cercare di mantenere la seconda milestone per garantire il servizio di produzione per i Data Challenges degli esperimenti Alberto Masoni 14

15 Sites in LCG 2/EGEE 0 : Situazione 4 Giugno 2004 Roma-1 (in manutenzione il 4-6) Tutta INFN-GRID e ora inserita in EGEE-0 Alberto Masoni Countries 58 Sites (45 Europe, 2 US, 5 Canada, 5 Asia, 1 HP) Coming: New Zealand, China, other HP (Brazil, Singapore) 3800 cpu

16 What is missing? (functionally) A full storage element dcache has had many problems Packaged to be deployed Is dcache sufficient/the only solution? Demonstrated integration of Tier 1 MSS s Alberto Masoni 16

17 I DATA CHALLENGES DEGLI ESPERIMENTI ALICE Partito in marzo terminata fase I (simulazione) interoperabilità di 3 grid attualmente in corso fase 2 (ricostruzione) fase 3 (analisi) prevista con prototipo EGEE/ARDA ATLAS In partenza Interoperabilità LCG GRID3 Nordugrid CMS Appena concluso Enfasi su data transfer LHCb Appena partito Utilizzo di Dirac e LCG Prestazioni soddisfacenti Problemi su data transfer 3 3 esperimenti su su 4 4 avranno i i Data Data Challenges in in corso corso a a partire dai dai prossimi giorni giorni Alberto Masoni 17

18 PIANIFICAZIONE FASE 2 E in corso la pianificazione per la fase 2 del progetto ( ) Gli esperimenti stanno effettuando una revisione del loro modello di calcolo e delle loro stime delle capacità richieste per il Tier 0 i Tier 1 i Tier 2 Sta partendo un attività di pianificazione per i Tier 1 Alberto Masoni 18

19 Phase 2 Planning Outline June 2003 Establish editorial board for LCG TDR September 2004 Consolidated Tier 1, Tier 2 Regional Centre Plan Background for the draft MoU October C RRB Revised version of basic computing models Revised estimates of overall Tier 1, Tier 2 resources Current state of commitments of resources Regional Centres Experiments High level plan for ramping up the Tier 1 and large Tier 2 centres Prepared by planning team including experiments, LCG management and regional centre representatives October 2004 C RRB Draft MoU End 2004 Initial computing models agreed by experiments April 2005 C RRB Final MoU End June 2005 TDR Alberto Masoni 19 Agreed by by POB POB 3 3 June June 04 04

20 CONCLUSIONI Il progetto europeo EGEE e partito Continuita con EDG/EDT Sinergia con LCG nel management, nelle scelte sul middleware, nell infrastruttura Evoluzione con rapide release e, allo stesso tempo continuità del servizio basato su LCG 2 Primo prototipo glite disponibile Il servizio LCG 2 è partito ed è in uso per i Data Challenge degli esperimenti Il sistema ha permesso (e permette) l uso di ingenti risorse, distribuite worldwide Il contributo italiano è il più rilevante Il CNAF rappresenta di gran lunga il contributo piu importante Grid IT (l insieme degli altri centri italiani) ha fornito un Tier 1 equivalente Resta ancora molto da fare Punto critico: Data Management Problemi sul data transfer Pianificazione risorse di storage (disco/cpu insufficiente) E cominciata la pianificazione per la fase 2 Alberto Masoni 20