My research combines the foundations and applications of data science, drawing on disciplines of computer science, computer networking, algorithms, machine learning, probability, and statistics to create new methods that are inspired by challenges in application domains. My major current interests include:
- Algorithms for sampling massive graph streaming data
Data analytics and algorithms for network measurement and resilience
Data science applications in engineering, currently transportation and hydrology
- 2014 to present: Texas A&M University, Department of Electrical and Computer Engineering, Full Professor (with tenure)
- 2015 to present: Director, Texas A&M Engineering Big Data Initiative
Other Current Affiliations and Professional Service
- 2015 to present: Professor by Courtesy, Texas A&M Department of Computer Science and Engineering
- 2015 to present: Associate Member, Oxford-Man Institute of Quantitative Finance
- 2015-2017: Member of Board of Directors, ACM Sigmetrics
- 2014 to present: Chief Editor for Big Data, Frontiers in ICT
- 2014 to present: Editor-at-Large, IEEE/ACM Transactions on Networking
Previous Positions and Education
- 2013-2014: Rutgers University / DIMACS, NJ: Research Professor
- 1995-2013: AT&T Labs-Research, NJ: Distinguished Member of Technical Staff (final level) and AT&T Fellow.
- 1986-1995: Post-doc and faculty positions in Dublin, Ireland and Heidelberg, Germany.
- 1983-1986: Queen Mary College, University of London, PhD, Mathematical Physics (1987)
- 1979-1983: Christ’s College, University of Cambridge, BA Natural Sciences (1982) and MMath (Part III Maths, 1983), MA (1986)
- Spring 2016: ECEN 489/689 Data Mining and Analysis
- Fall 2014, Fall 2015: ECEN 689 Data Science for Communications Networks
Research Interests and Collaborators
- Streaming Algorithms for Big Graph Sampling
- Rough Paths in Data Analytics
- Big Data in Transportation
- Data Science in Hydrology
- Measurement in Software Defined Networking
- Network Resilience through Data Analytics (multiple projects)
- Read more about current opportunities
Recent Research Funding
- NeTS: Small: Collaborative Research: Distributed Approximate Packet Classification, National Science Foundation, Award 1618030, 9/1/2016-8/31/2019. PIs Nick Duffield (Texas A&M) & Minlan Yu (Yale). Total $350,996 / Duffield $198,346.
- DEDUCE: Distributed Enclave Defense Using Configurable Edges. DARPA $11,500,000, 7/1/2015-6/30/2018, Applied Communication Sciences. Texas A&M Subaward, PI: Nick Duffield. Total $11,500,000 / Duffield $389,872
- Traffic Measurement from High-level Names in Software Defined Networking, Google Faculty Research Award, 2015, $79,992, PIs N.G. Duffield and Minlan Yu (University of Southern California).
- Approximation Methods for Massive Graph Analytics, Intel Corporation, 2016, PI N.G. Duffield, $30,000
- Understanding Multi-Scale Hydrology: Fusion of BIG DATA from Ground Networks and Space-Based Satellites, Texas A&M Big Data Seed Grant, January 2016, PI: Binayak Mohanty (Texas A&M Biological and Agricultural Engineering); co-PI: N. Duffield. Total $50,000./ Duffield $25,000
- Improving Understanding of Travel Behavior and Transportation Systems through Big Data Analytics, Texas A&M Big Data Seed Grant, January 2016, PI: Shawn Turner (Texas A&M Transportation Institute) co-PI: N. Duffield. Total $50,000 / Duffield $25,000
- Boosting Attack Identification through Correlated System & Network Monitoring. Texas A&M Cybersecurity Seed Grant, January 2016,. PI: N. Duffield, co-PI: Guofei Gu (Texas A&M Computer Science and Engineering). Total $50,000 / Duffield $25,000
Recent Short Term Research Visits
- 6/20/2016-7/15/2016, Intel Research, Santa Clara, CA
- 7/1/2015-7/31/2015, Oxford-Man Institute of Quantitative Finance
Selected Recent / Upcoming Talks
- 12/5/2016: Keynote, 2016 BigGraphs Workshop at IEEE BigData’16, Washington, DC
- 11/29/2016: Yale University, New Haven, CT
- 7/13/2016: Intel Research, Santa Clara, CA
- 7/8/2016: University of Southern California
- 1/21/2016: Informatics Institute, University of Florida
- 11/2015: ECE University of Texas at Austin
- 12/6/2014: Keynote, IPCCC, Austin TX
- 9/12/2014: Invited talk, Heilbronn Conference, Bristol, UK
- 8/2014: Sampling for Big Data: A Tutorial (with G. Cormode), ACM SIGKDD 2014
- 2/26/2014: Big Data Lecture Series, University of Illinois at Urbana Champaign
- 2/11/2014: Computer Engineering Eminent Scholar Seminar, Texas A&M University
- 12/10/2013: Facebook, Menlo Park CA
- 12/10/2013: Google, Mountain View CA
- 11/13/2013: Invited Speaker and Panelist, Big Data in the Mathematical Sciences, Warwick University
- 09/11/2013: Keynote Speaker, ITC 25, Shanghai
- Program Committee 2016 BigGraphs Workshop at IEEE BigData’16
- Scientific Programme Committee, 9th Bernoulli/IMS World Congress in Probability and Statistics, Toronto 2016
- Chief Editor for Big Data, Frontiers in ICT, since October 2014
- Editor-at-Large, IEEE/ACM Transactions on Networking, since September 2014
- TPC, International Workshop on Traffic Monitoring Analysis (TMA). Barcelona, April 2015
- TPC, IEEE International Conference on Network Protocols, ICNP 2014
- TPC, Workshop on MAthematical performance Modeling and Analysis, MAMA 2014
- TPC Co-Chair, IFIP Performance 2013,
- Publication list, most downloadable.
- Bibliographies: Google Scholar; DBLP; MathSciNet; USPTO
- H-Index: 58 (Google Scholar, > 12,800 citations )
- Erdös Number: 2
- ACM SIGMETRICS Test of Time, 2013, for Fast Accurate Computation of Large-Scale IP Traffic Matrices from Link Loads (2003), awarded jointly with Albert Greenberg, Matt Roughan and Yin Zhang. This award is given each year to a previous conference paper still making an impact 10-12 years after publication. Citation: “The paper presented a novel, remarkably fast, and accurate method for practical and rapid inference of traffic matrices in IP networks from link load measurements, augmented by readily available network and routing configuration information.”
- ACM SIGMETRICS Test of Time, 2012, for Network Tomography on General Topologies (2002), awarded jointly with Tian Bu, Francesco Lo Presti and Don Towsley. This award is given each year to a previous conference paper whose impact is still felt 10-12 years after publication. Citation: “This paper is a pioneering work in network tomography and it presented novel and formal approaches to perform tomography on networks under general setting, from which various performance, e.g., delay, packet lost,..etc., can be estimated.”
- AT&T Fellow, 2007. Citation: “For fundamental contributions to sampling, analysis and inference from network measurements that have had broad impact on AT&T and the industry.”
- IEEE Fellow, 2005. Citation: “For contributions to the measurement, analysis and management of telecommunications networks.”
Major research interests
- Big Graph Anayltics
- Data Analytics and Mobility Modeling in Cellular Networks
- Machine Learning in Network Management: Exploring the complex causal relationships between measurable network behavior and customer experience
- Optimal Sampling (including Smart Sampling of Flow Records): Developing efficient sampling algorithms that provably optimize trade-off between sample size and estimation variance, even for heavy-tailed distributions typical in the Internet. Structure aware sampling for more accurate range queries. Fair sampling over subpopulations. Versions currently deployed in network measurement infrastructure.
- Architectures for Fine Grained Traffic Measurement: Router level data structures for storage and retrieval of delay measurements. Enhancing performance measurement through temporal locality,
- Graph Sampling: unbiased node sampliing in P2P networks.
- Network Security and Attack Detection: stress testing traffic, machine learning of flow classifiers
- Performance Monitoring. Sampling for temporal loss patterns, now standardized in the IETF.
- Trajectory Sampling. Coordinate packet sampling across routers based on hash of invariant packet fields. Now standardized in the IETF.
- Network Tomography.
- Traffic Matrices: inferring traffic matrices from link level traffic aggregates. Deployed in network measurement infrastructure.
- Link Performance: inferring link level loss and delay by correlating end-to-end measurements across a mesh of paths. Support for reporting by transport protocols standardized in the IETF.
- The Hose Model for Virtual Private Networks: service without advance need of a traffic matrix
- Network Measurement Infrastructure: early generation (T3 speed) passive monitors
- Analysis and Measurement of Effective Bandwidths: using
- Statistical Mechanics
- Quantum dynamical semigroups and mean-field theory
- Large deviations and group representations
- Non-equilibrium phase transitions