Welcome to my personal webpage. I am a Full Professor at the Institute of Computing Technology (ICT), Chinese Academy of Sciences and University of Chinese Academy of Sciences. Additionally, I am the director of the Advanced Computer Systems Research Center, ICT, Chinese Academy of Sciences.

My research focuses on creating the unified discipline of Evaluation Science and Engineering, which I refer to as "Evaluatology." I have also coined the English term "Evaluatology" with contributions from my doctoral student, Hongxiao Li. I have made achievements in the field of high-performance computing and data center system software and successfully transferred 35 patents to leading enterprises. As mentioned in the prospectus of a listed company, I am the principal inventor of the patent "A Method and Intelligent Builder for Autonomous Construction of Cluster Operating System Kernel," along with two other patents and a software copyright (contributed by other contributors), which have been transferred to the company, accounting for 34.44% of its total capital.

Furthermore, I have founded or initiated the International Open Benchmark Council, Top 100 Science and Technology Evaluation Initiative, the international journal "BenchCouncil Transactions on Benchmarks, Standards, and Evaluations," as well as several international conferences.

Apart from my academic research and entrepreneurial endeavors, I am also a poet who finds great joy in life. In 2024, I will be publishing a poetry collection titled "Walking Together in the Wilderness" together with my daughter.

I have received various honors, including the National Science and Technology Progress Award (Second Class, 2005), the Outstanding Achievement Award of the Chinese Academy of Sciences (2006), the Best Paper Award at the IEEE International Symposium on Workload Characterization (IISWC) 2013, and the Outstanding Editor Award of the IEEE Transactions on Parallel and Distributed Systems (TPDS) in 2020.

You can reach me at the following contact information:
Office: Room 1038, Institute of Computing Technology, Chinese Academy of Sciences, No. 6 South Road, Zhongguancun, Haidian District, Beijing, China.
Email: jianfengzhan.benchcouncil@gmail.com

Evaluation is a crucial aspect of human existence and plays a vital role in various fields[1]. However, it is often approached in an empirical and ad-hoc manner, lacking consensus on universal concepts, terminologies, theories, and methodologies. I formally introduce the discipline of evaluatology, which encompasses the science and engineering of evaluation. Derived from the essence of evaluation, I propose five axioms focusing on key aspects of evaluation outcomes as the foundational evaluation theory. These axioms serve as the bedrock upon which a universal framework for evaluation is proposed, encompassing concepts, terminologies, theories, and methodologies that can be applied across various disciplines. Building upon the science of evaluation, I propose a formal definition of a benchmark as a simplified and sampled evaluation condition that guarantees different levels of equivalency. This concept serves as the cornerstone for a universal benchmark-based engineering approach to evaluation across various disciplines, which I refer to as benchmarkology.

Under my leadership, the big data and AI benchmarking projects, BigDataBench [2,3] and AIBench [4,5], have had significant impacts. BigDataBench is widely adopted in academia and industry worldwide, with the article published at HPCA'14 receiving 726 citations.

I am the founding chair of the International Open Benchmark Council (BenchCouncil) (https://www.benchcouncil.org ). Renowned scholars such as Matsuoka, Satoshi, Andrew A. Chien, Jeffrey S. Vetter, and John Shalf have ranked BenchCouncil's contributions alongside SPEC, founded in 1988, and before MLCommons, which was founded by Google and other industry and academic leaders such as Stanford and Facebook [6]. In their citation, they state, "Other commonly used benchmarking sets have been curated by organizations such as the Standard Performance Evaluation Corporation (SPEC), the International Open Benchmark Council (BenchCouncil), or MLCommons—with proxies for CPUs, distributed systems, accelerators, artificial intelligence—or have been developed by companies such as Intel (HiBench), Baidu (DeepBench), and Google (PerfKit)."

[1] Jianfeng Zhan, Lei Wang, Wanling Gao et al. Evaluatology: The Science and Engineering of Evaluation. arXiv preprint arXiv:2404.00021, 2024.
[2] Wang, Lei, Jianfeng Zhan, et al. "Bigdatabench: A big data benchmark suite from internet services." In 2014 IEEE 20th international symposium on high-performance computer architecture (HPCA), pp. 488-499. IEEE, 2014. Google Citations: 711. The first author, Lei Wang, is my Ph. D student.
[3] Jia, Zhen, Lei Wang, Jianfeng Zhan, et al. "Characterizing data analysis workloads in data centers." IISWC 2013 Best Paper Award.
[4]Hao, Tianshu, …Jianfeng Zhan. "Edge AIBench: Towards comprehensive end-to-end edge computing benchmarking." In Benchmarking, Measuring, and Optimizing: First BenchCouncil International Symposium, Bench 2018, Seattle, WA, USA, December 10-13, 2018. Google Citations: 81. The first author, Tianshu Hao, is my Ph. D student.
[5] Tang, Fei, Wanling Gao, Jianfeng Zhan, et al. "AIBench training: Balanced industry-standard AI training benchmarking." In 2021 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 24-35. IEEE, 2021. Google Citations: 21. The first author, Fei Tang, is my Ph. D student.
[6] Matsuoka, Satoshi, Jens Domke, Mohamed Wahib, Aleksandr Drozd, Andrew A. Chien, Raymond Bair, Jeffrey S. Vetter, and John Shalf. "Preparing for the Future—Rethinking Proxy Applications." Computing in Science & Engineering 24, no. 2 (2022): 85-90.

I spearheaded the research and development of the cluster operating system [7] for the Dawning 4000A, which achieved the 10th position on the Top 500 supercomputer list in June 2004 (source: https://www.top500.org/system/173564/). As mentioned in Sugon's initial public offering prospectus, my team's patent titled "A Method for Autonomously Constructing Cluster Operating System Kernels and an Intelligent Constructor," along with two other patents and one software IP, were transferred to Sugon, accounting for 34.44% of its total equity share [8]. Sugon is China's leading supercomputer company. In recognition of my contributions, I received the Outstanding Science and Technology Achievement Prize from the Chinese Academy of Sciences in 2005 and the Second Prize of The State Scientific and Technological Progress Award from the Central People's Government of the People's Republic of China in 2006.

Furthermore, I played a pivotal role in the design and implementation of cutting-edge cloud computing systems, namely PhoenixCloud [9,11] and DawningCloud [10], on the Dawning 5000 and 6000 Supercomputers. These innovative solutions revolutionized the support for high-performance computing and the consolidation of heterogeneous workloads on cloud systems. Additionally, I transferred 34 OS patents to Huawei, a world-class communication and IT company.

[7] Jianfeng Zhan, and Ninghui Sun. "Fire Phoenix cluster operating system kernel and its evaluation." In 2005 IEEE International Conference on Cluster Computing, pp. 1-9. IEEE, 2005.
[8] Sugon' initial public offering prospectus. Page 1-1-62. https://pdf.dfcfw.com/pdf/H2_AN201410230007390902_1.pdf .Accessed at September 10, 2023
[9] Jianfeng Zhan, Lei Wang, Bibo Tu, Yong Li, Peng Wang, Wei Zhou, and Dan Meng. "Phoenix cloud: Consolidating heterogeneous workloads of large organizations on cloud computing platforms." The first workshop on Cloud Computing and its Applications (CCA'08), 2008. https://www.alcf.anl.gov/events/cca-cloud-computing-and-its-applications
[10] Wang, Lei, Jianfeng Zhan, Weisong Shi, and Yi Liang. "In cloud, can scientific communities benefit from the economies of scale?" IEEE Transactions on Parallel and Distributed Systems 23, no. 2 (2011): 296-303. Google Citations: 214. The first author Lei Wang is my Ph. D student at the publication.
[11] Jianfeng Zhan, et al. "Cost-aware cooperative resource provisioning for heterogeneous workloads in data centers." IEEE Transactions on Computers 62, no. 11 (2012): 2155-2168. Google Citations: 69.