Currently cloud infrastructures are mostly homogeneous -- composed of a large number of machines of the same type -- centrally managed and made available to the end user using the three standard delivery models: Infrastructure-as-a-service (IaaS), Platform-as-a-Service (PaaS) and Software-as-a-Service (SaaS). As clouds increase in size and as machines of different types are added to the infrastructure in order to maximise performance and power efficiency, heterogeneous clouds are being created. However, exploiting different architectures such as graphics processing units, many integrated cores and data flow engines, poses significant challenges.To efficiently access heterogeneous resources and, at the same time, to exploit these resources to reduce application development effort, to make optimisations easier and to simplify service deployment requires a reevaluation of our approach to service delivery. The evolving complexity of the cloud ecosystem will eventually render traditional cloud management techniques ineffectual. Self-organisation and self-management are powerful techniques for managing complexity. Some of our initial simulation results using self organisation and self optimisation indicate that these can be used as the basis of a new cloud management and delivery model capable of efficiently dealing with issues that arise at scale. Our preliminary work has centred on promoting access to power efficient heterogeneous resources by shifting the deployment and optimization effort from the consumer to the software stack running on the cloud infrastructure. With CloudLightning, we propose to extend this work and to build a cloud management and delivery infrastructure based on these principles. Given the prohibitive expense associated with empirical experimentation on hyperscale cloud infrastructures, data gathered on our testbed will be used to simulate this infrastructure and to evaluate the self organisation approach in that context.

Extreme Data is an incarnation of Big Data concept distinguished by the massive amounts of data that must be queried, communicated and analyzed in (near) real-time by using a very large number of memory/storage elements and Exascale computing systems. Immediate examples are the scientific data produced at a rate of hundreds of gigabits-per-second that must be stored, filtered and analyzed, the millions of images per day that must be mined (analyzed) in parallel, the one billion of social data posts queried in real-time on an in-memory components database. Traditional disks or commercial storage cannot handle nowadays the extreme scale of such application data. Following the need of improvement of current concepts and technologies, ASPIDE’s activities focus on data-intensive applications running on systems composed of up to millions of computing elements (Exascale systems). Practical results will include the methodology and software prototypes that will be designed and used to implement Exascale applications. The ASPIDE project will contribute with the definition of a new programming paradigms, APIs, runtime tools and methodologies for expressing data-intensive tasks on Exascale systems, which can pave the way for the exploitation of massive parallelism over a simplified model of the system architecture, promoting high performance and efficiency, and offering powerful operations and mechanisms for processing extreme data sources at high speed and/or real-time.