PixFeedAnImagePubSubSystem.docxVIP

PixFeed An Image PubSub System Shyam Naren Kandala Anjaneya Srujan Narkedamalli Karan Mehta Computer Engineering Introduction Motivation A reliable cloud based pub-sub system to deliver images based on content of image Users can subscribe to images based on descriptive tags Eg: “train”. Users can upload images, select from suggested tags or provide custom tags. Similar to Instagram but subscription based on image content rather than publisher. Use case: Subscribe to all images from an ongoing event Architecture   System Components Amazon S3 cloud storage is used for storing the images. S3 is integrated with Amazon CloudFront CDN for faster delivery of images. Apache Kafka is message broker for publish subscribe mechanism Backend server: REST APIs built in Node.JS Android application for user interface Clarifai API for suggesting tags based on image recognition Implementation User uploads the image using PixFeed mobile app (Image available through S3 URL) Users can share images by selecting auto suggested tags or providing custom tags Image URL pushed on Kafka topic corresponding to a image tag Users can subscribe to image feeds based on tags Images delivered based on Kafka topic consumption offset Challenges Configuration of initial system Libraries for Kafka-Node.js interconnection have lot of external dependencies Difficult to test on Windows Platform As a solution, we used Cloud9, online IDE with Linux shell Partitioning and Replication of topics Multiple libraries available for Kafka-Node.js interconnection, but none of them support this functionality Possible through shell commands As an alternative, we spawned a child process that ran a shell command Performance Evaluation Application Server Specifications 512 MB RAM 2 GB Disk Space Load testing with JMeter Produce API Response time Create topic API Response time Subscribe and fetch API Response time Kafka choking point Effects of scaling Improvements with CDN PixFeed Demo PixFeed Demo part 1: /open?id=0B1z