A Thread-Based Implementation of the Message Passing Interface (MPI-1) Standard
Description
Introduction
AzequiaMPI is a thread-based full implementation of the Message Passing Interface (MPI-1) standard. It is being built through the knowledge gained in the study of the code and design of MPICH2, Open MPI and other implementations, like TOMPI, TMPI, MPC-MPI, and several more. The most inportant characteristic of the implementation is that built every MPI node as a thread, and not as an operating system process.
AzequiaMPI has three main layers:
- The kernel, called IDSP or Azequia, provides point to point communication and RPC-based group and thread management. Implements two versions, a BLK kernel implementing a concurrency model based on POSIX 1003.1c threads primitives, and a LFQ kernel, implementing synchronization through non-blocking lock-free structures. The kernel can be used directly from applications, avoiding MPI interface.
- INET, or Network Interface, that implements network facilities. It is under development, based on the study and experiences with the Open MPI BTLs. Our goal is provide TCP/IP and Infiniband facilities.
- MPI Interface, upon two layers above, provides MPI semantics to applications.
AzequiaMPI uses other tools:
- Process Manager Interface (from MPICH2) through MPD (Multi-Purpouse Daemon). It is used for launching applications in cluster environments.
- HWLOC (HardWare Locality) from Open MPI project. It is used for binding threads to processors.
- Etc.
Design model
AzequiaMPI implements an MPI endpoint as a thread, launching a container process in each shared memory machine. Figure shows current implementation models of MPI (P is process with a rank, T is a process with a rank and th is a thread without MPI weight).
Process-based MPI |
Hybrid model |
Thread-based MPI |
Current work
1. Improving several aspects of the implementation:
- Collective operations
- Static Variables issues through pre-compiling and other techniques
- Datatype and packing in shared memory thread communication
- Adding MPI-2 features supported by threads (e.g. One Sided Communication)
- Profiling tools
2. Adding network (TCP/IP and Infiniband) support to the implementation.
3. AzequiaMPI is now being ported to a TCP/IP FPGA cluster on top of hardware cores PowerPC.
Platforms
AzequiaMPI runs on different platforms:
- Linux based clusters.
- Heterogeneous DSPs/FPGAs multicomputers from Sundance company.
- Xilkernel/Petalinux operating systems on top of Microblaze software processor in Xilinx FPGAs.
- Texas Instruments C6000 DSPs on top of DSP/BIOS. For implementing Pthread semantic upon DSP/BIOS semaphores we developed OSI (Operating System Interface).
>You can get latest versions from the download section.
AzequiaMPI performance has been compared to other implementations, process and thread based. Latency and bandwidth in shared memory take advance of thread common address space. A set of figures are presented below, showing snapshots of the development.
Microbenchmarks performance
July 2011. We are studying one-copy mechanisms for communicating processes in shared memory multicore computers, and algorithms from some implementations, in an Intel Nehalem dual socket quad-core E5620 (shared memory) with 12 MBytes cache size. We explore SMARTMAT, LiMIC2, and KNEM under Open MPI and MPICH2, as well as other thread based implementations like MPC-MPI.
Figures show results from initial research in latency and bandwidth for point to point communication and some collectives with one-to-all and all-to-one patterns. Note that we use binding for ranks to processors in round robin (rank i to core i, not SMT), collectives are executed with 8 ranks. Note as well different color order in figures:
 |
 |
 |
 |
 |
 |
We are working in designing and implementation of new algorithm for taking advance of thread in shared memory for collective communication.
Euroben Matrix Product Performance
July 2011. Next figures show Euroben benchmark performance measurements. Euroben benchmark does a matrix product using different communication mechanisms provided by MPI. Benchmark gives some weight to communication time, so expect an improvement for implementations of MPI with higher communication performance.
All tests are run with 8 ranks in a Intel Nehalem (E5620) two-socket quad-core machine, with 12 MB of L3 cache. All ranks are bind to processors (rank i to processor i without SMT).
Next figures shows a matrix product using standard communication (MPI_Send/MPI_Recv) of data between tasks. Also relative performance compared to AzequiaMPI is showed. X-axis shows the square matrix side size. A table with bytes sended for each matrix side size is provided.
|
 |
 |
Next figures shows a matrix product where data between tasks is communicated by MPI broadcast collective operation. Relative performance compared to AzequiaMPI is showed as well. All parameters are the same as figures above.
 |
 |
HPL and NAS benchmarks performance
July 2011. Next figures show realistic performance measurements through High Perfomance Linpack (HPL) and NAS Parallel Benchmarks (NPB). Low communication weight in these benchmarks result in no virtual differences between implementations.
Both benchmarks are run in an Intel Nehalem dual-socket quad-core machine (E5620) with 12 MB of L3 cache. 8 tasks run with binding of tasks to processors (rank i to processor i without SMT).
Althought we choose NPB SP multizone benchmark, flat MPI is used, with 8 ranks and no OpenMP threads.
HPL run with following parameters: N=20.000, NB=16, PxQ=2x4 and 1-ring broadcast.
 |
 |
Additional information
We are interested in High Performance Computing software research and development. The International Exascale Software Project group has defined a roadmap for coordinating efforts from the international open source software community to create a software environment to exploit exascale systems in next years. We are interested in aspects from this roadmap and their development inside our projects.
1. We are studying influence of software in power consumption. Our goal is find and apply software techniques for saving energy in supercomputers. Initially, we are working in evaluating blocking and busy waiting in synchronization and communication, using respectively BLK and LFQ versions of the AzequiaMPI kernel. We are collaborating with CenitS/Computaex in this field.
2. We are interested in applications using MPI. Our goal is to improve AzequiaMPI performance and scalability for supporting real world applications efficiently in different platforms.
3. Some HPC useful links are:
a) PRACE. Partnership for Advanced Computing in Europe
b) HPC-Europe2. Pan-Eurpean Research Infraestructure for High Performance Computing
c) IESP. International Exascale Software Project
d) PlanetHPC. Research and Invertigation Roadmap for High Performance Computing in Europe
 |
 |
 |
 |
Documentation
Manuals
We are writting an User and Installtion Guide for AzequiaMPI. Coming soon ...
Related papers
Díaz Martín, J.C., Gómez-Tostón Gutiérrez, Carolina , Cortés Fácila, Álvaro , Rico Gallego, J.A."Una experiencia con MicroBlaze como procesador del sistema". I Simposio en computación empotrada, SiCE 2010 (CEDI 2010). Actas del congreso, ISBN: 978-84-92812-69-1 Pags.: 185-192. Valencia, España. September 2010.
Download
Díaz Martín, J.C., Gómez-Tostón Gutiérrez, Carolina , Cortés Fácila, Álvaro , Rico Gallego, J.A. "Issues on Building an MPI Cluster on Microblaze". 2010 International Conference on ReConFigurable Computing and FPGAs (Reconfig 2010). Conference Proceedings. ISBN 978-0-7695-4314-7. Pags.: 220-225. Cancún, México. December 2010.
Download
Díaz Martín, J.C., Rico Gallego, J.A., Álvarez Llorente, J.M., Perogil Duque, F.J. "An MPI-1 Compliant Thread-based Implementation". Lecture Notes in Computer Science. Recent Advances in Parallel Virtual Machine and Message Passing Interface. Springer-Verlag, pp. 327-329. 16th EuroPVM/MPI. Espoo, Finland. September 2009.
Download
Rico Gallego, J.A., Álvarez Llorente, J.M., Díaz Martín, J.C., Perogil Duque, F.J. "A Network Service for DSP Multicomputers" Lecture Notes in Computer Science. Springer Berlin / Heidelberg. Volume 5022/2008. ISSN 0302-9743 (Print) 1611-3349 (Online) Algorithms and Architectures for Parallel Processing. Springer Berlin / Heidelberg. Pags. 169-172. ISBN 978-3-540-69500-4 Berlin, 2008.
Download
Antúnez Gómez, P.P., Perogil Duque, F.J., Rico Gallego, J.A., Díaz Martín, J.C. "Aspectos de implantación de Azequia en nodos FPGA de multicomputadores Sundance™" Proceeding of VIII Jornadas de Computación Reconfigurable y Aplicaciones, ISBN: 978-84-612-5635-8 Madrid (Spain), September 18-19, 2008
Download
Rico Gallego, J.A., Álvarez Llorente, J.M., Perogil Duque, F.J., Antúnez Gómez, P.P., Díaz Martín, J.C. "A Pthreads-based MPI-1 Implementation for MMU-less Machines" Proceedings of 2008 International Conference on Reconfigurable Computing and FPGAs (Reconfig 2008), IEEE Computer Society Press, ISBN: 978-84-612-5635-8, pp 277-282 Cancun (Mexico), December 3-5, 2008
Download
Díaz Martín, J.C., Rico Gallego, J.A., Álvarez Llorente, J.M. "Efficient MPI Implementation upon DSP/BIOS on Sundance", Multi-computersProceedings of Second International Symposium on Communications, Control and Signal Processing (ISCCSP 2006) Ed. Suvisoft. ISBN: 2-908849-17-8 Marrakech, Morocco, 13-15 March 2006.
Download
Rico Gallego, J.A., Díaz Martín, J.C., Álvarez Llorente, J.M. "An MPI Implementation for Distributed Signal Processing" 12th European PVM/MPI Users’ Group Meeting (EuroPVM/MPI 2005) Lecture Notes in Computer Science LNCS Volume 3666 / 2005 Springer-Verlag, pp. 475-483.ISSN: 0302-9743 , ISBN: 3-540-29009-5 Editors: Beniamino Di Martino, Dieter Kranzlmüller, Jack Dongarra. Sorrento, Italy, September 18-21, 2005.
Download
Díaz Martín, J.C., Rico Gallego, J.A., Álvarez Llorente, J.M., Calvo Jurado, C. "On Interface Design for Distributed Signal Processing", Proceeding of 12th European Signal Processing Conference, EUSIPCO 2004, pp. 1365-1368Vienna, Austria, September 6-10, 2004 ISBN: 3-200-00165-8 Ed. SuviSoft LTD. F. Hlawatsch, G. Matz, M. Rupp, B. Wistawel.
Download
Díaz Martín, J.C., Rodríguez García, J.M., Álvarez Llorente, J.M., García Zapata, J.L., Rico Gallego, J.A. "A Distributed Framework For Signal Processing", Proceeding of IADIS International Conference Applied Computing 2004 Lisbon, Portugal, March 2004.
Download
García Zapata, J.L., Díaz Martín, J.C., Rico Gallego, J.A. "Handy Speech Recognition Development on DSK Platforms" Paper accepted at European DSP Education and Research Symposium (EDERS 2004) NEC, Birmingham, UK, November 16, 2004.
Rico Gallego, J.A., Díaz Martín, J.C., Rodríguez García, J.M., Álvarez Llorente, J.M., García Zapata, J.L. "Porting P4 to Digital Signal Processing Platforms", Lecture Notes in Computer Science series, pp. 362-368 (Proceedings of 10th European PVM/MPI User’s Group Meeting) Springer Verlag, Berlin, Germany,Venice, Italy, September 29 - October 2, 2003.
Download
Álvarez Llorente, J.M., Díaz Martín, J.C., Rodríguez García, J.M., García Zapata, J.L., Rico Gallego, J.A. "IDSP: un framework distribuido para multicomputadores DSP TMS320C6000", Proceedings of XIV Jornadas de Paralelismo, pp. 385-389 Leganés, Madrid, September 9-11, 2003 ISBN 84-8409-159-7.
Download
Download
License
Please, read the license information before download AzequiaMPI.
Releases
You can download latest version under development from the svn repository: http://gim.unex.es/svn/azqmpi/trunk. It can be accessed by web at: http://gim.unex.es/websvn/azqmpi. If you want to get involved and apply changes, please, send an This email address is being protected from spambots. You need JavaScript enabled to view it. to get an account.
Please, if you have any comments or any proposal about AzequiaMPI or related software, please, contact us by forum or by This email address is being protected from spambots. You need JavaScript enabled to view it. .
A wiki has been created (most in Spanish) for internal documentation about development and using platforms and software.
The latest stable release for Linux clusters can be downloaded from here:
| DATE | VERSION | FILE | README | CHANGES | ||
| 2011, July 31th | 2.0.3 | AzequiaMPI
|
Readme.txt | Changelog.txt | ||
| 2010, Mar 8th |
1.4.0 | AzequiaMPI
|
Readme.txt | Changelog.txt |
We have developed a version on FPGAs upon PetaLinux/Microblaze:
| DATE | VERSION | FILE | README | CHANGES | ||
| 2010, Dec 10th | 2.1.2_fpga | Azequia
|
Readme.txt | Changelog.txt | ||
| 2010, Dec 10th |
1.4.0_fpga | AzequiaMPI
|
Readme.txt | Changelog.txt |
We have ported AzequiaMPI to DSP multicomputers. A version for a multicomputer SMT310Q from Sundance Company, with DSPs C64x (SMT361, SMT361A, etc.) of Texas Instruments is available in the Azequia page.
Disclaimer
Neither the University of Extremadura, nor any of their employees, makes any
warranty express or implied, or assumes any legal liability or responsibility
for the accuracy, completeness, or usefulness of any information, apparatus,
product, or process disclosed, or represents that its use would not infringe
privately owned rights.
