CEA/DAM DSSI open source software
ARCANE
Arcane is a platform for development and services designed and used since 2000 as part of the Simulation program. Arcane helps the development of physics codes and optimize performance on large parallel computers, regardless of the complexity of the data structure. Thanks to an advanced object-oriented design, Arcane presents flexibility and modularity in the construction of simulation software, with different levels of reading. It offers the actors of the development process (doctors, numericians and computer scientists) the opportunity to focus on their core business. This approach facilitates, in a short time, the development of innovative scientific software.
Auks is a client-server application that adds support for Kerberos V tickets to a Batch-type application. The Auks server part provides optimized Kerberos ticket storage and ticketing services to ensure performance. adequate in HPC environment. The client part of Auks is built on top of an API allowing access in C language to the proposed functionalities. Auks comes with an example plugin adding Kerberos support to the Open Source Resource Manager Slurm, developed by the Lawrence Livermore National Laboratory (LLNL), version 2.2.
Bridge is an application that provides computing center users with a unified method of accessing resources: interactive and batch job submission, description of execution constraints, launch of parallel executions, monitoring and / or shutdown of executions on computers, etc…
ClusterShell is an event-based Python library to execute commands in parallel to a large number of distant cluster nodes, servers or workstations. The library defines and implements the notion of node group needed for large systems and provides several convenient command-line tools.
Environment Modules a development in Tcl language of the "module" command. It makes it easy to manage the software environment of users working on UNIX / LINUX by providing them with all the software and tools they need.
FRAME
The FRAME library, for FRAme field MEshing, is a library that provides quadrilateral mesh generation algorithms in dimension 2, and hexahedral or hexahedral dominant in dimension 3. These algorithms in the R & D state, are intended to be made available for production.
The proposed algorithms are based on the notion of frame fields. Their field of application is restricted to geometric models of the CAD type (with sharp edges) in dimension 2 and to all geometric domains in dimension 3.
GLoST (Greedy Launcher Of Small Tasks) is a lightweight, highly scalable tool for launching independent non-MPI processes in parallel.
GMDS
The Generic Mesh Data Structure (GMDS) library is a C ++ library that can model and manipulate 2D and 3D meshes in both a sequential context and a distributed context (high-performance computing).
GMDS makes it possible to represent any type of cells of dimension 2 (triangles, quadrilaterals, arbitrary polygons) and of dimension 3 (tetrahedra, hexahedra, quadrangular pyramids, triangular and pentagonal prisms, arbitrary polyhedra).
A special effort was made on the memory occupancy of these cells: it is minimized while remaining in a generic context of use. The proposed algorithms are based on the notion of fields of orientation (frame fields in English). Their field of application is restricted to geometric models of the CAD type (with sharp edges) in dimension 2 and to all geometric domains in dimension 3.
Hairgap is a set of tools to transfer data over a unidirectional network link (typically a network diode).
HERCULE
These services focus on protection / recovery, post-processing (including visualization) and data exchange between several simulation codes.
HP2P (Heavy Peer To Peer) a MPI based benchmark for network diagnostic. A GUI tool is provided to easily post-processing the results.
IVRE (Instrument de veille sur les réseaux externes) or DRUNK (Dynamic Recon of UNKnown networks) is a network recon framework, including tools for passive recon (flow analytics relying on Bro, Argus, Nfdump, fingerprint analytics based on Bro and p0f and active recon).
JCHRONOSS is a tool used as a validation driving force, particularly suited for parallel environments. Its aim is, from a given lists of tasks, to distribute this workload over the machine used to start the validation. The integrity and coherency of tasks is maintained over the whole validation. A task can be many things. A single task is started with `sh -c` calls and support a lot of commands. Thus, a test can represent any command that can be launched from a shell. As the majority of validation are started from a shell, this technique allows us to be as generic as possible.
Log analysis software which is able to assign automatic actions when certain identified situations appear in the logs according to rules defined by the user.
Kernel Linux – module 9P
The linux kernel is the base of the operating system, and is the standard interface for accessing hardware, network and files. Understanding the kernel is important both for the developments in which the CEA participates (Luster file system, 9P & RDMA protocols) and to analyze more effectively the problems that may be encountered during operation or implementation. calculators.
KVSNS implements a POSIX file system (UNIX standard programming interface for data access) from the services of a KVS (Key Value Store), makes a deposit of values addressed by keys, and a manager object storage. KVSNS relies on REDIS, which is a common KVS (Remote Dictionary Server) in the open source community, and on the POSIX file system. It can also use the KVS and object storage provided by the Seagate MERO commercial product, which is at the heart of the SAGE project.
LCAP is a software for processing and distributing message flows. It allows modes of diffusion more flexible and more powerful than that proposed directly by Luster for the description of the modifying operations brought to the system of files.
LOVE
In a context of massive exploitation of large volumes of data, Love is a software for visualizing data from numerical simulation codes. It allows users to explore and analyze data from numerical simulations with display and retrieval services on 2D and 3D geometry. These large volumes of data can reach several billion mesh.
LUSTRE (contribution)
Luster software is a powerful file management system, particularly well suited to managing large data streams and volumes controlled by super computers. This file system (File system) manages the organization of information stored on multiple storage devices (hard disks, magnetic tapes, ...), especially on tens of thousands of disks His concept of parallel file system allows between other to obtain high capacity and expandability of storage volume, speed of execution and fault tolerance. Its high performance makes it possible to meet the needs of the most demanding simulations in terms of storage, especially when it comes to adapting to a considerable production of data.
MALP
MALP (Multi-Application on-Line Profiling) is a profiling software consisting of analyzing the execution of a software application in order to know its behavior at runtime.
Code profiling allows you to control when running a software :
The list of functions called and the time spent in each of them,
The use of the processor (or the heart of the processor),
-
The list of functions called and the time spent in each of them,
-
The use of the processor (or the heart of the processor),
-
Memory usage ... etc.
The analysis of the behavior of one or more applications is done through analysis reports. Such reports are intended to help developers better understand their programs and aim to make measurable the performance criterion in order to positively influence the applications throughout development.
Miasmis a free and open source (GPLv2) reverse engineering framework. Miasm aims to analyze / modify / generate binary programs. Here is a non exhaustive list of features: Representing assembly semantic using intermediate language Emulating using JIT (dynamic code analysis, unpacking, ...) Expression simplification for automatic de-obfuscation ...
MilkCheck is a Python-based distributed, highly parallel and flexible service manager. It runs commands across various servers, based on dependencies between them, and offers a compact execution summary. It aims to manage service starting and checking on very large number of servers, like in HPC world. It can run tens of thousands of commands across thousand servers in very short time.
Mooshika is a user space library of functions designed to facilitate the integration of the Remote Direct Memory Access (RDMA) transport layer into a file server. This RDMA protocol therefore allows memory transfer to memory data via the interconnection network.
La bibliothèque MPC (Multi-Processor Computing) is a library dedicated to parallel programming in the high-performance computing context. MPC offers programming interfaces comparable to those currently used in parallel computing: MPI (Message Passing Interface) and POSIX Thread.
NABLA
Nabla is a compiler source sources; that is to say a translator of the language (of the same name) specialized for the numerical analysis towards different languages for a diversity of architectures and machines. Nabla makes it possible to write a computer program adapted to numerical analysis, thanks among other things to the use of mathematical operators.
NFS-Ganesha The NFSv4 server functions integrally in user space mode. It has modules which enable it to access various namespaces. The current version supports:
-
Generic access to a file system by means of the API POSIX.
-
Access to SNMP: information is viewed in /proc mode, each variable being assigned to a file.
-
A backend module makes the NFS-GANESHA server a proxy NFSv4.
-
The FUSE interface enables NFS export via NFS-GANESHA for products having a FUSE interface.
ODACE
Odace is a modular software for analyzing data from numerical simulation codes. In a context of massive exploitation of large volumes of data, it allows users to access and exploit the data of their calculations through extraction services, combinations and mathematical treatments. It also offers a graphical service that presents the results obtained in the form of curves.
PCOCC is a tool allowing users of a computer with distributed architecture (cluster) to simply instantiate virtual machines in order to execute parallel applications there. This gives users complete control of their runtime environment and can customize it to their needs. The tool integrates with the resource manager so that it can host virtual machines on cluster nodes in competition with other compute tasks. The virtual machines of a user are interconnected by a private network and isolated from the network of the computer so as not to disturb its operation.
PADAWAN
The purpose of the Parallel Data Workflow for Analysis (PaDaWAn) software is to relieve the calculation codes of data transfer operations. It builds on the existing I / O interface and transports the transfer operations to separate IT resources, freeing the simulation software from the random file system (filesystem). His intervention, at the level of the information transmitted during the execution of the code and during their transfer, ranks it among the solutions in transit. By playing both the role of writer for one and reading data for others, PaDaWAn allows a series of applications without recourse to storage units.
PHOBOS
Phobos is a software that makes available, under an object interface, distributed storage resources of a very varied nature (disk, flash, SSDs, magnetic tapes, other object managers). It provides robust, scalable, long-term archiving systems for our future computing center architectures.
Robinhood This software for monitoring, audit and purging of large file systems, enables in particular:
-
Generation of detailed reports on the usage profile of a file system.
-
Raising of alerts when the objects of the file system fulfil certain given conditions.
-
Application of complex purge policies defined by the administrator.
-
Careful control and balancing of the occupation rate of discs in a Lustre file system.
SELFIE (Self and Light proFIling Engine) is a lightweight profiling tool for HPC codes.
Shine is a command line tool designed to setup and manage Lustre file systems on a cluster. It aims at centralize and simplify Lustre filesystem management task with:
-
One human-readable configuration file to describe your filesystem,
-
Start, stop, format, repair (and more) your filesystem from a unique machine, thanks to a simple command line interface.
SIGMUND
Sigmund is a tool dedicated to performing non-regression tests. Entirely written in script language for reasons of simplicity and maintainability, it provides an environment in which it is easy to insert a new test. Sigmund is used as a launcher for non-regression testing for the NFS-Ganesha product.
SNECPLAYER
The SNECPLAYER software allows to stereoscopically display (relief vision) films composed of series of images of several million pixels on a wall of images consisting of a set of projectors, each controlled by a different computer (CPU + GPU). The particularity of this tool is to be able to cut in parallel images in as many parts as there are projectors to feed and synchronize the display of images between the projectors. The tool consists of two separate software packages: the first one running in parallel supports the display of images and the second is the control user interface.
SPACE9
At the expiry of the EXA 1 generation (202x), the CEA supercomputers will use proxies I / O servers based on 9P and RDMA protocols (new concept of I / O delegator). These will have client implementation in the kernel space but not in user space. The Space 9 library in user space for access to 9P and liop file systems, via TCP or RDMA, serves two purposes:
SSHPROXY is a program used on a gateway to transparently connect users to internal machines via the SSH protocol. It supports the tools scp, sftp, rsync and the graphic offset. It offers a load balancing mechanism for connecting to internal machines, customizing group or user configuration, and advanced auditing features.
Sibyl is a Miasm-based tool that allows searching and identifying known functions in binaries by analyzing their edge effects.
This software relies on Miasm to start a sandbox machine, incorporates the necessary elements (allocate a string of characters for example), then places the arguments in the expected order taking into account the 'ABI (Application Binary Interface) targeted. The "sandbox" machine then starts to execute the code, and if the end of the function is reached without malfunction, it is examined whether the edge effects are those expected. If so, we can consider that the probabilities are high of having the expected function.
WI4MPI (Wrapper Interface For MPI) is a library which translates MPI calls to calls MPI another MPI library during the execution of a program: a code compiled with wi4mpi can be executed with any MPI implementation.
