Abstract: In this paper, we present Real-Time Flow Filter (RTFF) —a system that adopts a middle ground between coarse-grained volume anomaly detection and deep packet in-spection. RTFF was designed with the goal of scaling to high volume data feeds that are common in large Tier-1 ISP networks and providing rich, timely information on ob-served attacks. It is a software solution that is designed to run on off-the-shelf hardware platforms and incorporates a scalable data processing architecture along with lightweight analysis algorithms that make it suitable for deployment in large networks. RTFF also makes use of state of the art machine learning algorithms to construct attack models that can be used to detect as well as predict attacks.
About Authors: Abhrajit Ghosh, is Director of the Intelligent Information Assurance Systems Research Group at Applied Communication Sciences, USA. His research interests include network security, network management and data privacy. He holds two Master’s degrees, one in Computer Applica-tions from Delhi University, India and another in Computer Science from SUNY Binghamton, USA.
Yitzchak M. Gottlieb, is a Senior Research Scien-tist at Applied Communication Sciences, USA. His areas of interest are computer networking, operating systems, systems management and the interactions between those areas. He is a member of the IEEE, the ACM, and an associate member of Sigma Xi.
Aditya Naidu, is a Senior Research Scientist at Applied Communication Sciences. He has expertise in the telecommunication domain as a researcher, systems engineer, and software devel-oper. He received his M.S. degree in Telecommunication Networks from Polytechnic Institute of NYU and B.S. degree in Electronic and Telecommunication Engineering from Mumbai University.
Akshay Vashist, is a Senior Research Scientist in the Knowledge-Based Systems Department at Applied Communication Sciences, USA. He received his Ph.D. degree in Computer Science from Rutgers University, USA. His research interests include machine learning, pattern recognition and intelligent systems. He co-developed the LUPI (Learning Using Privileged Information) paradigm in collaboration with Vladimir Vapnik. He has published more than 25 refereed papers in major conferences and journals and has several patents.
Alex Poylisher, has developed solutions to ap-plied problems in multiple areas of systems and network management (including host and network security), wireless network transport, and simula-tion/emulation tools and testbeds (including host and network virtualization) since 1996. He has a Ph.D. degree in Computer Science from the University of Warwick, UK.
Ayumu Kubota, received the B.E. and M.E. degrees in Information Science from Kyoto University, Japan, in 1993 and 1995, respectively. He joined KDD (now KDDI) in 1995, and has been engaged in the re-search on mobile computing and secure communication system. He is currently a senior manager of Network Security Laboratory in KDDI R&D Laboratories Inc., Japan.
Yukiko Sawaya, received the B.E. and M.E. degrees in Electronic Engineering from Tohoku University, Japan, in 2004 and 2006, respectively. She joined KDDI in 2006 and has been engaged in research on network security. She is currently a research engineer of Network Security Laboratory in KDDI R&D Laboratories Inc., Japan.
Akira Yamada, received his Ph.D. degree in Information Science from Tohoku University, Japan and B.E. and M.E. degrees in Electrical and Electronic Engineering from Kobe University, Japan, in 1999 and 2001, respectively. He joined KDDI in 2001, and has been engaged in research on information security and network security.
Cite this article:
Abhrajit Ghosh,Yitzchak M. Gottlieb,Aditya Naidu et al. Managing High Volume Data for Network Attack Detection Using Real-Time Flow Filtering[J]. China Communications, 2013, 10(3): 56-66.
Arbor Networks. Worldwide Infrastructure Security Report[R], 2011 Vol. VII.
DHARMAPURIKAR S, KRISHNAMURTHY P, SPROULL T, et al. Deep Packet Inspection Using Parallel Bloom Filters[C]// Proceedings of the 11th Symposium on High Performance Interconnects: August 20-22, 2003. Stanford, CA, USA, 2003: 44-51.
VAPNIK V. Statistical Learning Theory[M]. Wiley-Interscience, 1998.
HAAG P. Watch Your Flows with NfSen and NFDUMP[C]// Proceedings of 50th RIPE Meeting: May 2-6, 2005. Stockholm, Sweden.
KARRENBERG D, ROSS G, WILSON P, et al. Development of the Regional Internet Registry System[J]. Cisco Internet Protocol Journal, 2001, 4(4): 17-29.
GREENE B R. Remote Triggering Black Hole Filtering. Cisco Systems[EB/OL]. http://www. cisco.com/public/cons/isp/essentials/Remote_Triggered_ Black_Hole_Filtering-02. pdf (2002).
MARTY R. Applied Security Visualization[M]. Addison-Wesley, 2009.