CSR: Moving MapReduce into the Cloud: Flexibility, Efficiency, and Elasticity (NSF CNS-1422119, 10/2014-09/2018)
Project description and goals
MapReduce, a parallel and distributed programming model on clusters of commodity hardware, has emerged as the de facto standard for processing large data sets. Although MapReduce provides a simple and generic interface for parallel programming, it incurs several problems including low cluster resource utilization, suboptimal scalability and poor multi-tenancy support. This project explores and designs new techniques that let MapReduce fully exploit the benefits of flexible and elastic resource allocations in the cloud while addressing the overhead and issues caused?by server virtualization. It broadens impact by allowing a flexible and cost-effective way to perform big data analytics. This project also involves industry collaboration, curriculum development, and provides more avenues to bring women, minority, and underrepresented students into research and graduate programs.
The research project is exectued in a cutting-edge lab located in the new science and engineering building. The server room is furnished with cutting-edge HP data center blade facility that has three racks of HP ProLiant BL460C G6 blade server modules and a 40 TB HP EVA storage area network with 10 Gbps Ethernet and 8 Gbps Fibre/iSCSI dual channels. It has three APC InRow RP Air-Cooled and UPS equipments for maximum 40 kWs in the n+1 redundancy design.
- Dr. Xiaobo Zhou, The Principal Investigator
- Dr. Jia Rao, The Co-Principal Investigator
- Wei Chen, Research Assistant (2015 - )
- Tiago Perez, PhD student (2016 - 2017)
- AiDi Pi, PhD student (2017 - )
- Yanfei Guo, PhD student (2014 - 2015)
- Dazhao Cheng, PhD student (2014 - 2016)
- “Characterizing Scheduling Delay for Low-latency Data Analytics Workloads”, Wei Chen, Aidi Pi, Shaoqi Wang, and Xiaobo Zhou, Proc. of the 32nd IEEE International Parallel and Distributed Processing Symposium (IPDPS), Vancouver, May 2018.
- “Performance Isolation of Data-intensive Scale-out Applications in Multi-tenant Clouds”, Palden Lama, Shaoqi Wang, Xiaobo Zhou, and Dazhao Cheng, Proc. of the 32nd IEEE International Parallel and Distributed Processing Symposium (IPDPS), Vancouver, May 2018.
- “Preemptive, Low Latency Datacenter Scheduling via Lightweight Virtualization”, Wei Chen, Jia Rao, and Xiaobo Zhou, Proc. of the USENIX Annual Technical Conference (ATC), 13 pages, Santa Clara, July 2017.
- “Addressing Memory Pressure in Data-Intensive Parallel Programs via Container based Virtulization”, Proc. of the 14th IEEE International Conference on Autonomic Computing (ICAC), Columbus, July 2017.
- “Addressing Performance Heterogeneity in MapReduce Clusters with Elastic Tasks”, Wei Chen, Jia Rao, and Xiaobo Zhou, Proc. of the 31st IEEE International Parallel and Distributed Processing Symposium (IPDPS), May 2017.
- “Improving Performance of Heterogeneous MapReduce Clusters with Adaptive Task Tuning”, Dazhao Cheng, Jia Rao, Yanfei Guo, Changjun Jiang, and Xiaobo Zhou, IEEE Transactions on Parallel and Distributed Systems, Vol. 28, No. 3, pages: 774-786, March 2017.
- “Moving Hadoop into the Cloud with Flexible Slot Management and Speculative Execution”, Yanfei Guo, Jia Rao, Changjun Jiang, and Xiaobo Zhou, IEEE Transactions on Parallel and Distributed Systems, Vol. 28, No. 3, pages: 774-786, March 2017.
- “Fault Tolerant MapReduce-MPI for HPC Clusters", Yanfei Guo, Wesley Bland, Pavan Balaji, and Xiaobo Zhou, Proc. of the 27th ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC), Austin Nov 2015.
- "Towards Energy Efficiency in Heterogeneous Hadoop Clusters by Adaptive Task Assignment", Dazhao Cheng, Palden Lama, Changjun Jiang, and Xiaobo Zhou, Proc. of the 35th IEEE International Conference on Distributed Computing Systems (ICDCS), 10 pages, Columbus, June/July 2015.
- "StoreApp: A Shared Storage Appliance for Efficient and Scalable Virtualized Hadoop Clusters”, Yanfei Guo, Jia Rao, Dazhao Cheng, Changjun Jiang, Cheng-Zhong Xu, and Xiaobo Zhou, Proc. of the 34th IEEE International Conference on Computer Communications (INFOCOM), 9 pages, Hong Kong, April 2015.
- "Resource and Deadline-aware Job Scheduling in Dynamic Hadoop Clusters", Dazhao Cheng, Jia Rao, Changjun Jiang, and Xiaobo Zhou, Proc. of the 29th IEEE International Parallel and Distributed Processing Symposium (IPDPS), 10 pages, Hyderabad, May 2015.
This material is based upon work supported by the National Science Foundation under Grant CNS-1422119. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).