Weiwei Yang 杨巍威

---

ABOUT ME

@Apache

• ASF Member
• Apache YuniKorn PMC member
• Apache Hadoop Committer & PMC member
• Apache Ozone Comitter

@Apple

• AIML data infrastructure

@Cloudera

• Apache Iceberg - next generation table format for Data Engineering and Data Warehouse.
• Next generation resource scheduling Apache YuniKorn (Incubating)
• Continue to evolve YARN for better enterprise adaption.
• Kubernetes, Containerize, performance, service, ML and more

@Alibaba

• Real-time computing with Apache Flink, Infra engineer
• Apache Hadoop YARN at 10k+ nodes scale. Focus area: resource over-subscription, multi-dimension resource support, global scheduling, placement constraint/node attributes.

@IBM

• Startup member of BigInsights project (A Hadoop based BigData Platform Offering by IBM)
• Distributed system backend engineer (cluster installation/monitoring)
• Team lead and BigData architect (worked with 50+ customers)
• Apache committer (Apache Hadoop HDFS & YARN, 50k+ LoC).

Education:

• Peking University master
• Wuhan University bachelor

Want to know more? See my portfolio.

My Talks/Posts:

1. Beyond Experimental: Spark on Kubernetes. KubeConf North America, Detroit. Oct, 2022.

2. Next Level Spark on Kubernetes with Apache YuniKorn (Incubating). ApacheConf 2021, Oct, 2021.

3. Efficient Spark Scheduling on K8s with Apache YuniKorn. ApacheConf 2020, Oct 2020.

4. YARN to Kubernetes: How to tackle the challenages of resource management. Hadoop Meetup Shanghai, Sep 26, 2020. download pdf

5. Cloud-Native Spark Scheduling with YuniKorn Scheduler. Spark & AI Summit, June 2020.

6. Energize Multi-tenancy Flink on K8s with YuniKorn . Virtual Flink Forward Conference, San Francisco, Apr 24, 2020

7. Next Generation Scheduling: In Hybrid Cloud Environment for High Performance and Optimized Execution of Mixed Workloads. Dataworks Summit, Washington D.C. May 23, 2019

8. Open Hybrid Architecture: Running Stateful Containers on YARN. Hortonworks Tech Blog. Dec 17, 2018.

9. Apache Hadoop YARN: State of the Union. ArchSummit, Beijing, China. slides download. Dec 07, 2018.

10. Apache YARN 3.x in Alibaba. Dataworks Summit, San Jose, CA. Jun 21, 2018.

11. Success at Apache: the Chance to Influence the World. ASF “Success at Apache” Blog Series. Jun 04, 2018.

12. Apache Spark and DB2 with BLU Acceleration: Making ‘People Flow’ in Cities Measurable and Analyzable. IBM Insights Conference, Las Vegas. NV_ Oct 2015.

13. Apache Hadoop Fundamentals. School of Computer Science and Technology. Aug 15, 2012.

Get connected with LinkedIn: Link me!.