IMPORTANT !
Main repo https://gitlab.liris.cnrs.fr/pagoda/pagoda is just here to propose a scaffolding repo to deploy PAGoDA. Its aims is not to store any deployment configuration.
PLEASE FORK THIS REPO TO STORE YOUR OWN CONFIGURATION.
PAGoDA Presentation
The PAGoDA project aims at facilitating the deployment of research oriented computational platforms that are based on cloud infrastructure and where numerical experimentation components made by researchers are containerized.
The PAGoDA project enables system administrators to describe a target computational platform that fits researchers specific needs and to quickly deploy the corresponding PAGoDA instance.
This repository targets platform administrators (or PAGoDA project contributors). PAGoDA instance users should refer to this documentation
User Documentation (TODO)
User documentation is not ready yet.
This link will be update when documentation will be ready.
Any deployed PAGoDA instance is based on four elements:
- an infrastructure composed of virtualization layers acting at different levels: cloud provider (OpenStack), containerization software (Docker), containerization orchestrator (Kubernetes), cluster orchestrator (Rancher),
- an API used by administrators to service the infrastructure and by researchers to submit their computational tasks and collect their results.
- a web client plugged over the API (and deployed within the infrastructure)
- a CLI client using the API (and installed on a local computer)
Because the infrastructure of any PAGoDA instance is conceived around container/cluster orchestration, it provides researchers with a dynamic experimentation environment: computational resources are allocated on demand, such resources can be concurently allocated and scale up is properly handled.
Additionnally, a PAGoDA instance offers user data space e.g. for experiments to store their results or logs, thus helping researchers in their artifacts management.
An indirect expected effect of the PAGoDA project is the generalisation of the usage of containerization for research oriented numerical proof of concepts, simple computational components, workflows or even full scale applications. In turn, this technical adoption should promote researchers work making it portable, repeatable and replicable for the community.
PAGoDA relies on multiple monitoring software, like the Elastic Stack (Beats, Logstach, ElasticSearch and Kibana), Prometheus and Grafana which helps:
- Developers for the maintenance of the hardware and software.
- Users when they track their experiments results.
PAGoDA is a project founded by the LIRIS.
PAGoDA architecture
Below is a schema showing the idea of what PAGoDA is and how PAGoDA interacts with the different software and hardware it uses.
For now, PAGoDA heavily relies on OpenStack in order to manage VMs, volumes and to deploy Kubernetes within them. So VMs are not represented in this schematic as every Kubernetes clusters rely on VMs within OpenStack. In general, a Kubernetes cluster is composed of at least 1 master node and 2 worker node but can be composed of more master and more worker (see Requirements for more details).
On the left side is shown resources accessible by researchers/users of a PAGoDA instance. They will have a Kubernetes clusters for their experiments. They will be able to track the status of their experiments with the User Monitoring Namespace in which we will provide them an Elastic Stack. Finally, their data will be stored in OpenStack Volumes to access them later, once their experiments are completed.
To be able to run their experiments, researchers/users will use PAGoDA API through PAGoDA WebUI or PAGoDA CLI.
On the right side is shown resources accessible by the administrators of PAGoDA. These resources are composed of a Kubernetes Cluster which host the PAGoDA services and highly available monitoring tools composed of Elsastic Stack, Prometheus and Grafana. These monitoring tools will not store researchers/users data experiments neither their logs. They will store logs of all Kubernetes Clusters status as well as logs from the VMs which hosts the Kubernetes Cluster. Administrators will also have access to a Rancher Kubernetes Cluster and to an admin section within PAGoDA API to help them manage all the Kubernetes Cluster, the researchers/users experiments, etc. Finally, all logs and administrators datas will be save in OpenStack Volumes.