_scale-a-near-real-time-ai-system-by-4x-and-beyond-with-apache-spark-yan-li-conviva-and-shubo-liu-conviva.jpg "Scale a Near Real-Time AI System by 4X and Beyond with Apache Spark Yan Li (Conviva) and Shubo Liu (Conviva) from worker nodes Video Screenshot Preview")
GalleryPlay Video:
(Note: The default playback of the video is HD VERSION. If your browser is buffering the video slowly, please play the REGULAR MP4 VERSION or Open The Video below for better experience. Thank you!)
⏲ Duration: 16 min 84 sec
✓ Published: 13-Jun-2018
✓ Published: 13-Jun-2018
Description:
During last year’s Spark Summit, we presented a near real-time spark based application for video streaming quality analysis. Building a new system is always sweet with a lot of fun, productizing it sometimes can be bitter. At our scale, it was worth spending development time to reduce machine requirements.nnWe did this by: optimizing the number of splits; minimizing intra-job data shuffling; and customizing our SerDe. With these optimizations we were able to use 40% fewer machines. The Spark U
During last year’s Spark Summit, we presented a near real-time spark based application for video streaming quality analysis. Building a new system is always sweet with a lot of fun, productizing it sometimes can be bitter. At our scale, it was worth spending development time to reduce machine requirements.nnWe did this by: optimizing the number of splits; minimizing intra-job data shuffling; and customizing our SerDe. With these optimizations we were able to use 40% fewer machines. The Spark U
Share with your friends:
Whatsapp | Viber | Telegram | Line | SMS
Email | Twitter | Reddit | Tumblr | Pinterest
Related Videos
_black-desert-new-player-node-and-worker-guide-124-afk-income.webp "EvilDoUsHarm")
_cloud-migrations-with-cribl-cloud.webp "Cribl’s suite of products help you gain the control and confidence you need to successfully migrate to the cloud. With routing, shaping, enriching, and search functionalities, data becomes more manageable, and allows you to work more efficiently. By routing data from existing sources to multiple destinations, you can ensure data parity in your new cloud destinations, before turning off your on-premises (or legacy) analytics, monitoring, storage or database products and tooling. Cribl can reduc")
_worker-nodes-in-kubernetes-explained.webp "Farhad")
_kubernetes-101-nodes.webp "Sean Wingert")
_black-desert-overview-of-my-nodes-and-worker-empire-for-valencia-meals-timber-crates-and-more.webp "EvilDoUsHarm")
⏲ 14 minutes 24 seconds 👁 11K
_black-desert-how-to-set-up-worker-nodes.webp "mmstargaming")
_5-filipinos-hurt-in-singapore-airlines-emergency-landing.webp "Five Filipinos sustained injuries while onboard a Singapore-bound flight from London buffeted by severe air turbulence. The Department of Migrant Workers says all five are in stable condition as of Wednesday evening, May 22.<br/><br/>Full story: https://www.rappler.com/nation/overseas-filipinos/injuries-singapore-airlines-emergency-landing-may-22-2024/")
_kubernetes-master-amp-worker-nodes-in-ubuntu-22-04-scripts-to-install-with-no-internet-connection.webp "La Rebelion Labs")
⏲ 34 minutes 34 seconds 👁 390
_black-desert-new-player-node-and-worker-guide-124-afk-income.jpg)
_cloud-migrations-with-cribl-cloud.jpg)
_worker-nodes-in-kubernetes-explained.jpg)
_kubernetes-101-nodes.jpg)
_black-desert-overview-of-my-nodes-and-worker-empire-for-valencia-meals-timber-crates-and-more.jpg)
_black-desert-how-to-set-up-worker-nodes.jpg)
_5-filipinos-hurt-in-singapore-airlines-emergency-landing.jpg)
_kubernetes-master-amp-worker-nodes-in-ubuntu-22-04-scripts-to-install-with-no-internet-connection.jpg)
![[+]](/images/menu.gif){kind=small}