Accurate and real-time traffic measurement is becoming increasingly critical for large variety of applications including accounting, bandwidth provisioning and security analysis. Existing network measurement techniques, however, have major difficulty dealing with large number of flows in today’s high-speed networks and offer limited scalability with increasing link speeds. Consequently, the current state of the art solutions have to resort to conservative sampling of the traffic stream and/or to account for only a few frequent flows that often fail to provide accurate estimates of traffic features.
Fundamentally, traffic measurement involves counting number of packets that satisfy some criteria, commonly referred to as user query or a rule, over a period of time. The traffic is measured in terms of flows, where a flow refers to a set of packets that have the same n-tuple value in their header fields. Typical definitions of flow include the 6-tuple:{prt, tos, sip, spt, dip, dpt} where, prt is the protocol field, tos is type of service, sip and dip are the source and destination IP addresses and spt and dpt are the source and destination ports, respectively.
Traditionally, measurement schemes have operated by maintaining unique per-flow based counters on high-density storage media, followed by aggregation of selected counters to answer queries. This mechanism is illustrated in Figure-1. An inherent and increasingly-widening performance gap between high-density storage access time and network bandwidth coupled with a significant I/O overhead makes it difficult to operate the paradigm in real-time. As a result, traditional schemes do not offer scalable measurement solutions for high data rate networks and have to resort to sampling and/or offline processing. Cisco’sNetFlow is one such widely deployed sample-based traffic measurement solution.
The key issue with sample based solutions is measurement accuracy. We argue that a scalable solution for real-time and accurate measurement of traffic has to dispose of conventional per-flow based statistics collection. Instead, in this work, we propose a query-driven measurement methodology that works by profiling passing traffic according to the given query to collect information of interest in real-time as shown in Figure-2. Unlike existing techniques, our solution processes streaming traffic at link speeds and hence, does not compromise measurement accuracy due to sampling.
The focus of this work is in evaluating the query-driven measurement paradigm utilizing embedded systems. An initial prototype has already been developed on an FPGA platform. A high level diagram of the prototype is presented in Figure-3. The architecture consists of a hardware-software co-designed solution that contains a highly-parallel and scalable array of processing elements in hardware where the user queries can be dynamically and independently mapped. The mapping and interaction with the user is controlled through a control processor and glue logic interface. Further details of the prototype solution have recently been published in ANCS’08.
