Social Media's Role in Network Management in Big Data

Network Management in Big Data

In day today world social media and social networking has received much attention from every people, like almost everyone has a Facebook account. This is where huge amount of data is being processed every day, in fact every second where Social networks accounts for large amount of consumer "big data". The average global Internet user spends two and a half hours daily on social media, in this scenario just consider how much data is being generated every minute by every user. The leading social networking sites are handling this big data in efficient way, when it reaches a comparison stage there's no beating Facebook in driving traffic to publishers. According to the data form US news the world's largest social network directed 15.4% of traffic in the history ever. Twitter is processing 143,199 tweets per second universally. Facebook's "like" button is pushed 2.7 billion times every single day.

Stats of Social Media Traffic

All this data created and handling process leads to huge amount of network traffic and questions every network managers “How to Manage?”, this leads us towards the concept of “FlowComb”.

FLOWCOMB

FlowComb is a network management framework that helps Big Data processing applications, such as Hadoop. It helps those applications in achieving high utilization and low data processing times and also it predicts application network transfers. This is achieved by a centralized decision engine collects data movement information from agents and schedules upcoming flows on paths such that the network does not become congested. Network transfers and software defined networking are detected using domain Knowledge to update the network paths to support these transfers without creating congestion...

... middle of paper ...

...ocessing.

DESIGN

FlowComb improves job processing times and prevents network congestion in Hadoop MapReduce clusters by predicting network transfers and scheduling them dynamically on paths with sufficient available bandwidth.

FlowComb consists of three modules:

• Flow prediction,

• Flow scheduling

• Flow control.

Flow Prediction

FlowComb detects data transfers between nodes in a Hadoop cluster using domain knowledge about the interaction between Hadoop components.

Flow Scheduling

The scheduler receives periodically a list of current or pending data transfers (i.e., source and destination IPs and volume), detects if any of them creates congestion on their default path and if it does, schedules them on a new path.

Flow Control

To exploit the full potential of FlowComb, the switches in the network must be programmable from a centralized controller.