Weiwei Yang 杨巍威

---

ABOUT ME

Apache

• ASF Member
• Ex-VP of Apache YuniKorn & PMC member
• Ex-Chiar of CNCF Batch working group
• Apache Hadoop Committer & PMC member
• Apache Ozone Comitter

Apple

• AIML data infrastructure

Cloudera

• Apache Iceberg - next generation table format for Data Engineering and Data Warehouse.
• Next generation resource scheduling Apache YuniKorn (Incubating)
• Continue to evolve YARN for better enterprise adaption.
• Kubernetes, Containerize, performance, service, ML and more

Alibaba

• Real-time computing with Apache Flink, Infra engineer
• Apache Hadoop YARN at 10k+ nodes scale. Focus area: resource over-subscription, multi-dimension resource support, global scheduling, placement constraint/node attributes.

IBM

• Startup member of BigInsights project (A Hadoop based BigData Platform Offering by IBM)
• Distributed system backend engineer (cluster installation/monitoring)
• Team lead and BigData architect (worked with 50+ customers)
• Apache committer (Apache Hadoop HDFS & YARN, 50k+ LoC).

Education:

• Peking University master
• Wuhan University bachelor

Want to know more? See my portfolio.

My Talks/Posts:

• Ray at Scale: Apple’s Approach to Elastic GPU Management. RaySummit, San Francisco, 2024.
• Revolutionizing Kube Scalability Testing with KWOK. KubeConf North America, Chicago, 2023.
• Beyond Experimental: Spark on Kubernetes. KubeConf North America, Detroit. Oct, 2022.
• Next Level Spark on Kubernetes with Apache YuniKorn (Incubating). ApacheConf 2021, Oct, 2021.
• Efficient Spark Scheduling on K8s with Apache YuniKorn. ApacheConf 2020, Oct 2020.
• YARN to Kubernetes: How to tackle the challenages of resource management. Hadoop Meetup Shanghai, Sep 26, 2020. download pdf
• Cloud-Native Spark Scheduling with YuniKorn Scheduler. Spark & AI Summit, June 2020.
• Energize Multi-tenancy Flink on K8s with YuniKorn . Virtual Flink Forward Conference, San Francisco, Apr 24, 2020
• Next Generation Scheduling: In Hybrid Cloud Environment for High Performance and Optimized Execution of Mixed Workloads. Dataworks Summit, Washington D.C. May 23, 2019
• Open Hybrid Architecture: Running Stateful Containers on YARN. Hortonworks Tech Blog. Dec 17, 2018.
• Apache Hadoop YARN: State of the Union. ArchSummit, Beijing, China. slides download. Dec 07, 2018.
• Apache YARN 3.x in Alibaba. Dataworks Summit, San Jose, CA. Jun 21, 2018.
• Success at Apache: the Chance to Influence the World. ASF “Success at Apache” Blog Series. Jun 04, 2018.
• Apache Spark and DB2 with BLU Acceleration: Making ‘People Flow’ in Cities Measurable and Analyzable. IBM Insights Conference, Las Vegas. NV_ Oct 2015.
• Apache Hadoop Fundamentals. School of Computer Science and Technology. Aug 15, 2012.

Get connected with LinkedIn: Link me!.