Le risorse Open Source per la ricerca in ambito scientifico

Transcript

1 Le risorse Open Source per la ricerca in ambito scientifico Claudio Gheller CINECA, Consorzio Interuniversitario Science and Computers: a close link FROM OBSERVATIONS TO THEORY Computers are fundamental in any aspect of research Development of new instruments Control and management of instruments Data acquisition, telemetry Data management and compression Data reduction and analysis Numerical simulations Comparison between numerical simulations and observations/experiments 1

2 Science and Computers: huge requests Often scientific numerical applications requires enormous computational resources A typical high end run, today can need: 100 to 500 Gbyte of RAM memory (the memory of 100 to 500 pcs) More than computing hours (more or less 11.5 years) 10 to 100 Tbyte of resident disk space (thousand of times that of your pc) Special computing resources are needed!!! HPC resources Untill a decade ago, HPC for science was represented by: specialized pieces of hardware Specific (and propretary) management software (Operating Systems) Specific (and propretary) compilers and user libraries Today the trend has completely changed. HPC MEANS PARALLEL COMPUTING!!! 2

3 Hardware With the rise of the Beowulf clusters, a new generation of supercomputing systems bacame available, based on OFF-THE-SHELF products (but be careful ) Low cost HPC, but with some penalties BlueGene 3

4 Linux Linux is a Unix-like computer operating system. Linux is one of the most prominent examples of free software and open source development; its underlying source code can be modified, used, and redistributed by anyone, freely. The Linux kernel was first released to the public on 17 September 1991, for the Intel x86 PC architecture. The kernel was augmented with system utilities and libraries from the GNU project to create a complete and friendly operating system, which later led to the alternate term GNU/Linux. Linux is now packaged for different uses in Linux distributions, which contain the kernel along with a variety of other software packages tailored to requirements. A typical desktop Linux distribution comprises a Linux kernel, GNU tools and libraries, additional software, documentation, a window system, window manager, and a desktop environment. Most of the included software is free software/open-source software which is distributed by its maintainers both as precompiled binaries and in source code form, allowing users to modify and compile the original source code if they wish. Other software included with some distributions may be proprietary and may not be available in source code form. Linux distributions Slackware, one of the first Linux distributions, founded in 1993 Debian, a non-commercial distribution maintained by a volunteer developer community with a strong commitment to free software principles Ubuntu, a newly popular desktop distribution derived from Debian Red Hat, maintained by the American company of the same name, which also provides a community version in the form of Fedora CentOS, a distribution derived from the same sources used by Red Hat, Mandriva, a Red Hat derivative popular in France and Brazil, opensuse, originally derived from Slackware with the system management software borrowed from Red Hat, maintained by the company Novell Gentoo, a distribution targeted at power users, known for its FreeBSD Ports-like automated system for compiling applications from source code Knoppix, a LiveCD distribution that runs completely from removable media and without installation to a hard disk Linspire, a commercial desktop distribution based on Debian, 4

5 The GNU Project The GNU Project is a free software project, announced in 1983 by Richard Stallman. It initiated the GNU operating system, software development for which began in January The founding goal of the project was, in the words of its initial announcement, to develop "a sufficient body of free software [...] to get along without any software that is not free. Particularly interesting the GNU Compiler Collection, a fundamental instrumento for code developers, that includes front ends for C, C++, Objective-C, Fortran, Java, and Ada, as well as libraries for these languages (libstdc++, libgcj,...). Many libraries like MPICH, LAM-MPI, GSL, PETSc, FFTW Simulation Software Some of the most ambitious packages have the goal of describing the behaviour of complex systems For astrophysics, many codes are available open source: Simulation packages Type GADGET Enzo Pluto Fly FLASH CactusCode Tree N-Body + SPH AMR AMR Tree N-body AMR AMR 5

6 The Millenium simulation The Virgo consortium, an international group of astrophysicists from the UK, Germany, Japan, Canada and the USA has realized the largest and most realistic simulation ever of the growth of cosmic structure and the formation of galaxies and clusters. The "Millennium Simulation" was performed using the Gadget2 code and employed more than 10 billion particles of matter to trace the evolution of the matter distribution in a cubic region of the Universe over 2 billion lightyears on a side. It kept the principal supercomputer at the Max Planck Society's Supercomputing Centre in Garching, Germany occupied for more than a month. By applying sophisticated modelling techniques to the 25 Terabytes (25 million Megabytes) of stored output, Virgo scientists are able to recreate evolutionary histories for the approximately 20 million galaxies which populate this enormous volume Data management tools Several data management tools are available as free software: MySQL is a multithreaded, multi-user SQL (Strucured Query Language) database management system (DBMS) which has more than 10 million installations PostgreSQL is a free software object-relational database management system (ORDBMS), released under a BSD-style license. HDF5 a standard and portable data format plus libraries and utilities 6

7 Data analysis and visualization tools Plenty of data analysis and visualization tools and libraries are available as free software: E.g. Visualization Toolkit (VTK) is an open source, freely available software system for 3D computer graphics, image processing, and visualization used by thousands of researchers and developers around the world. VTK consists of a C++ class library, and several interpreted interface layers including Tcl/Tk, Java, and Python. Professional support and products for VTK are provided by Kitware, Inc. VTK supports a wide variety of visualization algorithms including scalar, vector, tensor, texture, and volumetric methods; and advanced modeling techniques such as implicit modeling, polygon reduction, mesh smoothing, cutting, contouring, and Delaunay triangulation. In addition, dozens of imaging algorithms have been directly integrated to allow the user to mix 2D imaging / 3D graphics algorithms and data. Data analysis and visualization tools: VisIVO VisIVO (which stands for Visualization Interface for the Virtual Observatory) is a visualization and analysis software for astrophysical data. VisIVO can handle both observational and theoretical data (i.e. produced by numerical simulations). It can be used both as a stand-alone application, that acts on local files, and as an interface to the Virtual Observatory framework, from which it can retrieve the data. The development of VisIVO is funded by the Italian Institute for Astronomy and Astrophysics (INAF) and by CINECA See: 7

8 Vantaggi del software libero * essendo possibile modificare liberamente il software, è possibile personalizzarlo ed adattarlo alla proprie esigenze * il codice sorgente è sottoposto ad una revisione da parte di moltissime persone, pertanto è più difficile che contenga bachi e malfunzionamenti. In ogni caso, è sempre possibile per chiunque tenere un indice pubblico dei problemi, in modo che gli utenti li conoscano * se viene scoperto un baco o una falla di sicurezza, la sua correzione di solito è molto rapida * essendo il sorgente liberamente consultabile, è molto difficile inserire intenzionalmente nel software backdoor, cavalli di Troia o spyware senza che questi vengano prontamente scoperti ed eliminati, come invece è accaduto per alcuni software commerciali * non esistendo standard proprietari, le cui specifiche sono normalmente segrete, è molto più facile costruire software interoperabile * la complessità e le dimensioni di alcuni software liberi (ad esempio, dei sistemi operativi) è tale che è necessario il supporto commerciale di un'azienda; il software libero si presta a creare nuove opportunità di business nel campo della formazione e del supporto, oltre che della eventuale personalizzazione del software. Limiti del software libero * essendo un lavoro volontario, lo sviluppo del software libero sarebbe più lento rispetto al software proprietario; * alcune tipologie di software, soprattutto di nicchia, non sarebbero disponibili come software libero; infatti i software di nicchia non avrebbero abbastanza utenti per la creazione di una comunità che supporti lo sviluppo del software. * lo sviluppo del software libero avrebbe una struttura anarchica, che porta a risultati incoerenti e ad una mancanza di uniformità e consistenza; * nonostante il codice sorgente sia liberamente disponibile, non tutti sono in grado di apportarvi modifiche, inoltre l'usabilità spesso non è ottimale per un utente inesperto; * anche se spesso è disponibile molta documentazione, è sovente incompleta o non aggiornata. Alcune di queste critiche sono talvolta frutto di un'errata comprensione del software libero. Molte persone tendono infatti a considerare il software libero come prodotto esclusivamente da volontari, mentre sono molti i casi in cui è semplicemente un modello di sviluppo adottato a livello aziendale. 8

9 Applicazioni commerciali del software libero * Un software libero non deve necessariamente essere sviluppato a titolo gratuito o a fondo perduto. Purché si rispettino i vincoli della licenza d'uso, è possibile vendere un software libero; * Vi sono alcune aziende che adottano il modello di sviluppo del software libero per i propri prodotti commerciali. Il ritorno economico in questo caso può derivare dalla fornitura di assistenza e di know-how. * Lo stesso software può venire licenziato sia come proprietario, sia come software libero. La versione libera talvolta dispone di meno funzionalità, o è limitata ad un numero ristretto di piattaforme. * Vi sono poi aziende che sono strutturate integralmente per la vendita e l'assistenza di un determinato software libero: esempi classici sono alcune distribuzioni di Linux, come Red Hat o SUSE. Queste aziende utilizzano come base il software sviluppato dalla comunità, aggiungendo una serie di tool di configurazione o sviluppo, curando gli aspetti più tecnici e dando agli utenti finali un'assistenza mirata. * Sfruttando le caratteristiche della licenza BSD, alcune aziende preferiscono invece partire da un software libero per sviluppare un prodotto non libero. Dynamical simulations: The difficult story... We want to follow the evolution of a physical system according to some basic rules (gravity, fluid dynamics, cooling...). Typically: Complex physics Plenty of data Complex algorithms Lots of communications Unbalanced behaviour Huge data files 9

10 Numerical simulations in astrophysics. Example 1 The solar system The sun, birth and evolution of the planets Code Particle based SPH (Smoothed Particle Hydrodynamics) Physics Gravity, fluid dynamics, radiative cooling Scales Spatial = 10 9 km (100 a.u.) Mass = kg (10 solar masses) Numerical simulations in astrophysics. Example 2 Cluster of stars birth and evolution of the stars, cluster evolution Code Particle based SPH (Smoothed Particle Hydrodynamics) Physics Gravity, fluid dynamics, radiative cooling Scales Spatial = km ( a.u.) Mass = kg (1000 solar masses) 10

11 Numerical simulations in astrophysics. Example 3 Galaxy formation birth and evolution of galaxies (stars,blue + gas,red) Code Particle based SPH (Smoothed Particle Hydrodynamics) Physics Gravity, fluid dynamics, radiative cooling Scales Spatial = km (10 pc) Mass = kg (10 11 solar masses) Numerical simulations in astrophysics. Example 4 Galaxy cluster formation birth and evolution of a cluster of galaxy. Gas behaviour. Code Gadget, N-Body + SPH Physics Gravity, fluid dynamics Scales Spatial = km (10 Mpc) Mass = kg (10 15 solar masses) 11