Discussion:
[dpdk-dev] [PATCH 0/7] Add Membership Library
(too old to reply)
Yipeng Wang
2017-08-22 00:19:46 UTC
Permalink
DPDP Membership Library provides an API that can be used by many DPDK
applications to conduct one or more set-membership tests (we mention some
possible use cases below, but interested readers can refer to
[1] for a wider survey of use cases).

The basic functionalities of the Membership Library include
inserting a new member, deleting an existing member, and querying the existence
of a member. The query result would indicate with high accuracy which specific
set this member belongs to among a group of sets.

The Membership Library is an extension and generalization of traditional filter
data structures [2,3], which maintain a space-efficient “set-summary”.
There are two advantages of using such a set-summary rather than operating on a
“full-blown” complete list of elements: firstly it has a much smaller storage
requirement than storing the whole list of elements, and secondly set membership
tests (or other operations) is much more efficient than searching through the
complete list of elements.

A membership test for an element will return the set this element belongs to if
found (or return "not-found") with high accuracy. If needed, the accuracy of the
membership tests could be further increased with larger storage space.
Set-summary is a fundamental data aggregation component that can be used in many
network applications. It is a crucial structure to address performance and
scalability issues of diverse applications including overlay networks, wild card
flow classification, web-caches, load balancing, connection tracking,
data-centric networks, flow table summaries, network statistics and traffic
monitoring. Our Proof of Concept (PoC) using set-summary to optimize flow lookup
in Open vSwitch (OvS) shows a speedup of about 2-3X.

This patch set implements two types of set-summaries, i.e., hash-table based
set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the
non-cache and cache modes. The non-cache mode can incur a small chance of
false-positives which is the case when the set-summary indicates a key belongs
to a given set while actually it is not. The cache mode can also have
false-negatives in addition to false-positives. False-negatives means the case
when the set-summary indicates a key does not belong to a given set while
actually it does. This happens because cache mode allows new key to evict
existing keys. vBF only has false-positives similar to the non-cache HTSS.
However, one can set the false-positive rate arbitrarily. HTSS's
false-positive rate is determined by the hash-table size and the signature size.

We sent out RFC previously:
http://dpdk.org/ml/archives/dev/2017-May/066599.html

[1] A Broder and M Mitzenmacher, “Network Applications of Bloom Filters: A
Survey,” in Internet Mathematics, 2005.

[2] B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,”
Communications of the ACM, 1970.

[3] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than
Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.


Yipeng Wang (7):
member: implement main API
member: implement HT mode
member: implement vBF mode
member: add AVX for HT mode
member: enable the library
test/member: add functional and perf tests
doc: add membership documentation

MAINTAINERS | 7 +
config/common_base | 5 +
doc/api/doxy-api-index.md | 3 +-
doc/api/doxy-api.conf | 1 +
doc/guides/prog_guide/img/member_i1.svg | 1613 ++++++++++++++++++++++++++++++
doc/guides/prog_guide/img/member_i2.svg | 36 +
doc/guides/prog_guide/img/member_i3.svg | 148 +++
doc/guides/prog_guide/img/member_i4.svg | 450 +++++++++
doc/guides/prog_guide/img/member_i5.svg | 163 +++
doc/guides/prog_guide/img/member_i6.svg | 332 ++++++
doc/guides/prog_guide/img/member_i7.svg | 399 ++++++++
doc/guides/prog_guide/index.rst | 14 +
doc/guides/prog_guide/member_lib.rst | 432 ++++++++
doc/guides/rel_notes/release_17_11.rst | 17 +
lib/Makefile | 2 +
lib/librte_eal/common/eal_common_log.c | 1 +
lib/librte_eal/common/include/rte_log.h | 1 +
lib/librte_member/Makefile | 50 +
lib/librte_member/rte_member.c | 357 +++++++
lib/librte_member/rte_member.h | 518 ++++++++++
lib/librte_member/rte_member_ht.c | 567 +++++++++++
lib/librte_member/rte_member_ht.h | 115 +++
lib/librte_member/rte_member_vbf.c | 309 ++++++
lib/librte_member/rte_member_vbf.h | 85 ++
lib/librte_member/rte_member_version.map | 15 +
lib/librte_member/rte_member_x86.h | 111 ++
mk/rte.app.mk | 2 +
test/test/Makefile | 3 +
test/test/test_member.c | 550 ++++++++++
test/test/test_member_perf.c | 643 ++++++++++++
30 files changed, 6948 insertions(+), 1 deletion(-)
create mode 100644 doc/guides/prog_guide/img/member_i1.svg
create mode 100644 doc/guides/prog_guide/img/member_i2.svg
create mode 100644 doc/guides/prog_guide/img/member_i3.svg
create mode 100644 doc/guides/prog_guide/img/member_i4.svg
create mode 100644 doc/guides/prog_guide/img/member_i5.svg
create mode 100644 doc/guides/prog_guide/img/member_i6.svg
create mode 100644 doc/guides/prog_guide/img/member_i7.svg
create mode 100644 doc/guides/prog_guide/member_lib.rst
create mode 100644 lib/librte_member/Makefile
create mode 100644 lib/librte_member/rte_member.c
create mode 100644 lib/librte_member/rte_member.h
create mode 100644 lib/librte_member/rte_member_ht.c
create mode 100644 lib/librte_member/rte_member_ht.h
create mode 100644 lib/librte_member/rte_member_vbf.c
create mode 100644 lib/librte_member/rte_member_vbf.h
create mode 100644 lib/librte_member/rte_member_version.map
create mode 100644 lib/librte_member/rte_member_x86.h
create mode 100644 test/test/test_member.c
create mode 100644 test/test/test_member_perf.c
--
2.7.4
Yipeng Wang
2017-08-22 00:19:47 UTC
Permalink
Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a
set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the
set-summary.

The results of the membership test is not 100% accurate. Certain
false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set-summary"
is memory efficient and fast on lookup.

This patch add the main API definition.

Signed-off-by: Yipeng Wang <***@intel.com>
---
lib/Makefile | 2 +
lib/librte_eal/common/eal_common_log.c | 1 +
lib/librte_eal/common/include/rte_log.h | 1 +
lib/librte_member/Makefile | 48 +++
lib/librte_member/rte_member.c | 357 +++++++++++++++++++++
lib/librte_member/rte_member.h | 518 +++++++++++++++++++++++++++++++
lib/librte_member/rte_member_version.map | 15 +
7 files changed, 942 insertions(+)
create mode 100644 lib/librte_member/Makefile
create mode 100644 lib/librte_member/rte_member.c
create mode 100644 lib/librte_member/rte_member.h
create mode 100644 lib/librte_member/rte_member_version.map

diff --git a/lib/Makefile b/lib/Makefile
index 86caba1..c82033a 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -108,6 +108,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += librte_reorder
DEPDIRS-librte_reorder := librte_eal librte_mempool librte_mbuf
DIRS-$(CONFIG_RTE_LIBRTE_PDUMP) += librte_pdump
DEPDIRS-librte_pdump := librte_eal librte_mempool librte_mbuf librte_ether
+DIRS-$(CONFIG_RTE_LIBRTE_MEMBER) += librte_member
+DEPDIRS-librte_member := librte_eal librte_hash

ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y)
DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
diff --git a/lib/librte_eal/common/eal_common_log.c b/lib/librte_eal/common/eal_common_log.c
index 0e3b932..90db6a3 100644
--- a/lib/librte_eal/common/eal_common_log.c
+++ b/lib/librte_eal/common/eal_common_log.c
@@ -279,6 +279,7 @@ static const struct logtype logtype_strings[] = {
{RTE_LOGTYPE_CRYPTODEV, "cryptodev"},
{RTE_LOGTYPE_EFD, "efd"},
{RTE_LOGTYPE_EVENTDEV, "eventdev"},
+ {RTE_LOGTYPE_MEMBER, "member"},
{RTE_LOGTYPE_USER1, "user1"},
{RTE_LOGTYPE_USER2, "user2"},
{RTE_LOGTYPE_USER3, "user3"},
diff --git a/lib/librte_eal/common/include/rte_log.h b/lib/librte_eal/common/include/rte_log.h
index ec8dba7..ce1a3d0 100644
--- a/lib/librte_eal/common/include/rte_log.h
+++ b/lib/librte_eal/common/include/rte_log.h
@@ -87,6 +87,7 @@ extern struct rte_logs rte_logs;
#define RTE_LOGTYPE_CRYPTODEV 17 /**< Log related to cryptodev. */
#define RTE_LOGTYPE_EFD 18 /**< Log related to EFD. */
#define RTE_LOGTYPE_EVENTDEV 19 /**< Log related to eventdev. */
+#define RTE_LOGTYPE_MEMBER 20 /**< Log related to membership. */

/* these log types can be used in an application */
#define RTE_LOGTYPE_USER1 24 /**< User-defined log type 1. */
diff --git a/lib/librte_member/Makefile b/lib/librte_member/Makefile
new file mode 100644
index 0000000..997c825
--- /dev/null
+++ b/lib/librte_member/Makefile
@@ -0,0 +1,48 @@
+# BSD LICENSE
+#
+# Copyright(c) 2017 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in
+# the documentation and/or other materials provided with the
+# distribution.
+# * Neither the name of Intel Corporation nor the names of its
+# contributors may be used to endorse or promote products derived
+# from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+
+EXPORT_MAP := rte_member_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_member/rte_member.c b/lib/librte_member/rte_member.c
new file mode 100644
index 0000000..560b754
--- /dev/null
+++ b/lib/librte_member/rte_member.c
@@ -0,0 +1,357 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <string.h>
+
+#include <rte_eal.h>
+#include <rte_eal_memconfig.h>
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_malloc.h>
+#include <rte_errno.h>
+
+#include "rte_member.h"
+#include "rte_member_ht.h"
+#include "rte_member_vbf.h"
+
+TAILQ_HEAD(rte_member_list, rte_tailq_entry);
+static struct rte_tailq_elem rte_member_tailq = {
+ .name = "RTE_MEMBER",
+};
+EAL_REGISTER_TAILQ(rte_member_tailq)
+
+
+void *
+rte_member_find_existing(const char *name)
+{
+ struct rte_member_setsum *setsum;
+ struct rte_tailq_entry *te;
+ struct rte_member_list *member_list;
+
+ member_list = RTE_TAILQ_CAST(rte_member_tailq.head, rte_member_list);
+
+ rte_rwlock_read_lock(RTE_EAL_TAILQ_RWLOCK);
+ TAILQ_FOREACH(te, member_list, next) {
+ setsum = (struct rte_member_setsum *) te->data;
+ if (strncmp(name, setsum->name, RTE_MEMBER_NAMESIZE) == 0)
+ break;
+ }
+ rte_rwlock_read_unlock(RTE_EAL_TAILQ_RWLOCK);
+
+ if (te == NULL) {
+ rte_errno = ENOENT;
+ return NULL;
+ }
+ return setsum;
+}
+
+void
+rte_member_free(void *ss)
+{
+ struct rte_member_setsum *setsum;
+ struct rte_member_list *member_list = NULL;
+ struct rte_tailq_entry *te;
+
+ if (ss == NULL)
+ return;
+ setsum = ss;
+ member_list = RTE_TAILQ_CAST(rte_member_tailq.head, rte_member_list);
+ rte_rwlock_write_lock(RTE_EAL_TAILQ_RWLOCK);
+ TAILQ_FOREACH(te, member_list, next) {
+ if (te->data == ss)
+ break;
+ }
+ if (te == NULL) {
+ rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+ return;
+ }
+ TAILQ_REMOVE(member_list, te, next);
+ rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ rte_member_free_ht(setsum);
+ break;
+ case RTE_MEMBER_TYPE_VBF:
+ rte_member_free_vbf(setsum);
+ break;
+ default:
+ break;
+ }
+ rte_free(setsum);
+ rte_free(te);
+}
+
+
+void *
+rte_member_create(const struct rte_member_parameters *params)
+{
+ struct rte_tailq_entry *te;
+ struct rte_member_list *member_list = NULL;
+ struct rte_member_setsum *setsum = NULL;
+ int ret;
+
+ if (params == NULL) {
+ rte_errno = EINVAL;
+ return NULL;
+ }
+
+ if ((params->key_len == 0)) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, MEMBER,
+ "Memship create with invalid parameters\n");
+ return NULL;
+ }
+
+ member_list = RTE_TAILQ_CAST(rte_member_tailq.head, rte_member_list);
+
+ rte_rwlock_write_lock(RTE_EAL_TAILQ_RWLOCK);
+
+ TAILQ_FOREACH(te, member_list, next) {
+ setsum = (struct rte_member_setsum *) te->data;
+ if (strncmp(params->name, setsum->name,
+ RTE_MEMBER_NAMESIZE) == 0)
+ break;
+ }
+ setsum = NULL;
+ if (te != NULL) {
+ rte_errno = EEXIST;
+ te = NULL;
+ goto error_unlock_exit;
+ }
+ te = rte_zmalloc("MEMBER_TAILQ_ENTRY", sizeof(*te), 0);
+ if (te == NULL) {
+ RTE_LOG(ERR, MEMBER, "tailq entry allocation failed\n");
+ goto error_unlock_exit;
+ }
+
+ /* Create a new setsum structure */
+ setsum = (struct rte_member_setsum *) rte_zmalloc_socket(params->name,
+ sizeof(struct rte_member_setsum), RTE_CACHE_LINE_SIZE,
+ params->socket_id);
+ if (setsum == NULL) {
+ RTE_LOG(ERR, MEMBER, "Create setsummary failed\n");
+ goto error_unlock_exit;
+ }
+ setsum->type = params->type;
+ setsum->socket_id = params->socket_id;
+ setsum->key_len = params->key_len;
+ setsum->num_set = params->num_set;
+ setsum->name = params->name;
+ setsum->prim_hash_seed = params->prim_hash_seed;
+ setsum->sec_hash_seed = params->sec_hash_seed;
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ ret = rte_member_create_ht(setsum, params);
+ break;
+ case RTE_MEMBER_TYPE_VBF:
+ ret = rte_member_create_vbf(setsum, params);
+ break;
+ default:
+ goto error_unlock_exit;
+ }
+ if (ret < 0)
+ goto error_unlock_exit;
+ RTE_LOG(DEBUG, MEMBER, "Creating a setsummary table with mode %u\n",
+ setsum->type);
+
+ te->data = (void *)setsum;
+ TAILQ_INSERT_TAIL(member_list, te, next);
+ rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+ return setsum;
+
+error_unlock_exit:
+ rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+ rte_member_free(setsum);
+ return NULL;
+}
+
+
+int
+rte_member_add(const void *ss, const void *key, MEMBER_SET_TYPE set_id)
+{
+ const struct rte_member_setsum *setsum = ss;
+
+ if (setsum == NULL || key == NULL)
+ return -EINVAL;
+ int ret = 0;
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ ret = rte_member_add_ht(setsum, key, set_id);
+ break;
+ case RTE_MEMBER_TYPE_VBF:
+ ret = rte_member_add_vbf(setsum, key, set_id);
+ break;
+ default:
+ return -EINVAL;
+ }
+ return ret;
+}
+
+
+int
+rte_member_lookup(const void *ss, const void *key,
+ MEMBER_SET_TYPE *set_id)
+{
+ const struct rte_member_setsum *setsum = ss;
+ if (setsum == NULL || key == NULL || set_id == NULL)
+ return -EINVAL;
+
+ int ret = 0;
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ ret = rte_member_lookup_ht(setsum, key, set_id);
+ break;
+ case RTE_MEMBER_TYPE_VBF:
+ ret = rte_member_lookup_vbf(setsum, key, set_id);
+ break;
+ default:
+ return -EINVAL;
+ }
+ return ret;
+}
+
+
+int
+rte_member_lookup_bulk(const void *ss, const void **keys, uint32_t num_keys,
+ MEMBER_SET_TYPE *set_ids)
+{
+ const struct rte_member_setsum *setsum = ss;
+
+ if (setsum == NULL || keys == NULL || set_ids == NULL)
+ return -EINVAL;
+
+ int ret = 0;
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ ret = rte_member_lookup_bulk_ht(setsum, keys, num_keys,
+ set_ids);
+ break;
+ case RTE_MEMBER_TYPE_VBF:
+ ret = rte_member_lookup_bulk_vbf(setsum, keys, num_keys,
+ set_ids);
+ break;
+ default:
+ return -EINVAL;
+ }
+ return ret;
+}
+
+int
+rte_member_lookup_multi(const void *ss, const void *key,
+ uint32_t match_per_key, MEMBER_SET_TYPE *set_id)
+{
+ const struct rte_member_setsum *setsum = ss;
+
+ if (setsum == NULL || key == NULL || set_id == NULL)
+ return -EINVAL;
+ int ret = 0;
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ ret = rte_member_lookup_multi_ht(setsum, key, match_per_key,
+ set_id);
+ break;
+ case RTE_MEMBER_TYPE_VBF:
+ ret = rte_member_lookup_multi_vbf(setsum, key, match_per_key,
+ set_id);
+ break;
+ default:
+ return -EINVAL;
+ }
+ return ret;
+}
+
+
+int
+rte_member_lookup_multi_bulk(const void *ss, const void **keys,
+ uint32_t num_keys, uint32_t max_match_per_key,
+ uint32_t *match_count, MEMBER_SET_TYPE *set_ids)
+{
+ const struct rte_member_setsum *setsum = ss;
+ if (setsum == NULL || keys == NULL || set_ids == NULL ||
+ match_count == NULL)
+ return -EINVAL;
+ int ret = 0;
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ ret = rte_member_lookup_multi_bulk_ht(setsum, keys, num_keys,
+ max_match_per_key, match_count, set_ids);
+ break;
+ case RTE_MEMBER_TYPE_VBF:
+ ret = rte_member_lookup_multi_bulk_vbf(setsum, keys, num_keys,
+ max_match_per_key, match_count, set_ids);
+ break;
+ default:
+ return -EINVAL;
+ }
+ return ret;
+}
+
+
+int
+rte_member_delete(void *ss, const void *key, MEMBER_SET_TYPE set_id)
+{
+ struct rte_member_setsum *setsum = ss;
+ if (setsum == NULL || key == NULL)
+ return -EINVAL;
+ int ret = 0;
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ ret = rte_member_delete_ht(setsum, key, set_id);
+ break;
+ case RTE_MEMBER_TYPE_VBF:
+ default:
+ return -EINVAL;
+ }
+ return ret;
+}
+
+
+void
+rte_member_reset(const void *ss)
+{
+ const struct rte_member_setsum *setsum = ss;
+ if (setsum == NULL)
+ return;
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ rte_member_reset_ht(setsum);
+ break;
+ case RTE_MEMBER_TYPE_VBF:
+ rte_member_reset_vbf(setsum);
+ break;
+ default:
+ break;
+ }
+}
diff --git a/lib/librte_member/rte_member.h b/lib/librte_member/rte_member.h
new file mode 100644
index 0000000..de44b1b
--- /dev/null
+++ b/lib/librte_member/rte_member.h
@@ -0,0 +1,518 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+
+
+/**
+ * @file
+ *
+ * RTE Membership Library
+ *
+ * The Membership Library is an extension and generalization of a traditional
+ * filter (for example Bloom Filter) structure that has multiple usages in a
+ * variety of workloads and applications. The library is used to test if a key
+ * belongs to certain sets. Two types of such "set-summary" structures are
+ * implemented: hash-table based (HT) and vector bloom filter (vBF). For HT
+ * setsummary, two subtype or modes are available, cache and non-cache modes.
+ * The table below summarize some properties of the different implementations.
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ */
+
+
+/**
+ * <!--
+ * +==========+=====================+================+=========================+
+ * | type | vbf | HT-cache | HT-non-cache |
+ * +==========+=====================+==========================================+
+ * |structure | bloom-filter array | hash-table like without storing key |
+ * +----------+---------------------+------------------------------------------+
+ * |set id | limited by bf count | [1, 0x7fff] |
+ * | | up to 32. | |
+ * +----------+---------------------+------------------------------------------+
+ * |usages & | small set range, | can delete, | cache most recent keys, |
+ * |properties| user-specified | big set range, | have both false-positive|
+ * | | false-positive rate,| small false | and false-negative |
+ * | | no deletion support.| positive depend| depend on table size, |
+ * | | | on table size, | automatic overwritten. |
+ * | | | new key does | |
+ * | | | not overwrite | |
+ * | | | existing key. | |
+ * +----------+---------------------+----------------+-------------------------+
+ * -->
+ */
+
+
+#ifndef _RTE_MEMBER_H_
+#define _RTE_MEMBER_H_
+
+
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+#include <stdio.h>
+#include <stdint.h>
+#include <rte_hash_crc.h>
+
+/** The set ID type that stored internally in hash table based set summary. */
+typedef uint16_t MEMBER_SET_TYPE;
+/** Invalid set ID used to mean no match found. */
+#define RTE_MEMBER_NO_MATCH 0
+/** Maximum size of hash table that can be created. */
+#define RTE_MEMBER_ENTRIES_MAX (1 << 30)
+/** Maximum number of keys that can be searched as a bulk */
+#define RTE_MEMBER_LOOKUP_BULK_MAX 64
+/** Entry count per bucket in hash table based mode. */
+#define RTE_MEMBER_BUCKET_ENTRIES 16
+/** Maximum number of characters in setsum name. */
+#define RTE_MEMBER_NAMESIZE 32
+
+/** @internal Primary hash function to calculate 1st independent hash. */
+#define MEMBER_PRIM_HASH(key, key_len, seed) \
+ (uint32_t)(rte_hash_crc(key, key_len, seed))
+/** @internal Secondary hash function to calculate 2nd independent hash. */
+#define MEMBER_SEC_HASH(key, key_len, seed) \
+ (uint32_t)(rte_hash_crc(key, key_len, seed))
+
+
+struct rte_member_setsum;
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Parameter struct used to create set summary
+ */
+struct rte_member_parameters;
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Define different set summary types
+ */
+enum rte_member_setsum_type {
+ RTE_MEMBER_TYPE_HT = 0, /**< Hash table based set summary. */
+ RTE_MEMBER_TYPE_VBF, /**< vector of bloom filters. */
+ RTE_MEMBER_NUM_TYPE
+};
+
+/** @internal compare function for different arch. */
+enum rte_member_sig_compare_function {
+ RTE_MEMBER_COMPARE_SCALAR = 0,
+ RTE_MEMBER_COMPARE_AVX2,
+ RTE_MEMBER_COMPARE_NUM
+};
+
+/** @internal setsummary structure. */
+struct rte_member_setsum {
+ enum rte_member_setsum_type type;
+ const char *name;
+ uint32_t key_len;
+ uint32_t socket_id; /* NUMA Socket ID for memory. */
+ uint32_t prim_hash_seed;
+ uint32_t sec_hash_seed;
+
+
+ /* hash table based */
+ uint32_t bucket_cnt;
+ uint32_t bucket_mask;
+ uint8_t cache;
+ enum rte_member_sig_compare_function sig_cmp_fn;
+
+ /* vector bloom filter*/
+ uint32_t num_set;
+ uint32_t bits;
+ uint32_t bit_mask;
+ uint32_t num_hashes;
+ void *table;
+};
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Parameters used when create the set summary table. Currently user can
+ * specify two types of setsummary: HT based and vBF. For HT based, user can
+ * specify cache or non-cache mode. Here is a table to describe some differences
+ *
+ */
+struct rte_member_parameters {
+ const char *name; /**< Name of the hash. */
+
+ /**
+ * User to specify the type of the setsummary from one of
+ * rte_member_setsum_type.
+ *
+ * HT based setsummary is implemented like a hash table. User should use
+ * this type when there are many sets.
+ *
+ * vBF setsummary is a vector of bloom filters. It is used when number
+ * of sets is not big (less than 32 for current implementation).
+ */
+ enum rte_member_setsum_type type;
+ /**
+ * If it is HT based setsummary, user to specify the subtype or mode
+ * of the setsummary. It could be cache, or non-cache mode.
+ * Set iscache to be 1 if to use as cache mode.
+ *
+ * For cache mode, keys can be evicted out of the HT setsummary. Keys
+ * with the same signature and map to the same bucket
+ * will overwrite each other in the setsummary table.
+ * This mode is useful for the case that the set-summary only
+ * needs to keep record of the recently inserted keys. Both
+ * false-negative and false-positive could happen.
+ *
+ * For non-cache mode, keys cannot be evicted out of the cache. So for
+ * this mode the setsummary will become full eventually. Keys with the
+ * same signature but map to the same bucket will still occupy multiple
+ * entries. This mode does not give false-negative result.
+ */
+ uint8_t iscache;
+
+ /**
+ * For HT setsummary, num_keys equals to the number of entries of the
+ * table. When the number of keys that inserted to the HT setsummary
+ * approaches this number, eviction could happen. For cache mode,
+ * keys could be evicted out of the table. For non-cache mode, keys will
+ * be evicted to other buckets like cuckoo hash. The table will also
+ * likely to become full before the number of inserted keys equal to the
+ * total number of entries.
+ *
+ * For vBF, num_keys equal to the expected number of keys that will
+ * be inserted into the vBF. The implementation assumes the keys are
+ * evenly distributed to each BF in vBF. This is used to calculate the
+ * number of bits we need for each BF. User does not specify the size of
+ * each BF directly because the optimal size depends on the num_keys
+ * and false positive rate.
+ */
+ uint32_t num_keys;
+
+
+ /**
+ * The length of key is used for hash calculation. Since key is not
+ * stored in set-summary, large key does not require more memory space.
+ */
+ uint32_t key_len;
+
+
+ /**
+ * num_set is only relevant to vBF based setsummary.
+ * num_set is equal to the number of BFs in vBF. For current
+ * implementation, it only supports 1,2,4,8,16,32 BFs in one vBF set
+ * summary. If other number of sets are needed, for example 5, the user
+ * should allocate the minimum available value that larger than 5,
+ * which is 8.
+ */
+ uint32_t num_set;
+
+ /**
+ * false_postive_rate is only relevant to vBF based setsummary.
+ * false_postivie_rate is the user-defined false positive rate
+ * given expected number of inserted keys (num_keys). It is used to
+ * calculate the total number of bits for each BF, and the number of
+ * hash values used during lookup and insertion. For details please
+ * refer to vBF implementation and membership library documentation.
+ *
+ * HT setsummary's false positive rate is in the order of:
+ * false_pos = (1/bucket_count)*(1/2^16), since we use 16-bit signature.
+ * This is because two keys needs to map to same bucket and same
+ * signature to have a collision (false positive). bucket_count is equal
+ * to number of entries (num_keys) divided by entry count per bucket
+ * (RTE_MEMBER_BUCKET_ENTRIES). Thus, the false_postivie_rate is not
+ * directly set by users.
+ */
+ float false_positive_rate;
+
+ /**
+ * We use two seeds to calculate two independent hashes for each key.
+ *
+ * For HT type, one hash is used as signature, and the other is used
+ * for bucket location.
+ * For vBF type, these two hashes and their combinations are used as
+ * hash locations to index the bit array.
+ */
+ uint32_t prim_hash_seed;
+
+ /**
+ * The secondary seed should be a different value from the primary seed.
+ */
+ uint32_t sec_hash_seed;
+
+ int socket_id; /**< NUMA Socket ID for memory. */
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Find an existing set-summary and return a pointer to it.
+ *
+ * @param name
+ * Name of the set-summary
+ * @return
+ * Pointer to the set-summary or NULL if object not found
+ * with rte_errno set appropriately. Possible rte_errno values include:
+ * - ENOENT - value not available for return
+ */
+void *
+rte_member_find_existing(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Create set-summary (SS).
+ *
+ * @param params
+ * parameters to initialize the setsummary
+ * @return
+ * return the pointer to the setsummary
+ * return value is NULL if the creation failed
+ */
+
+void *
+rte_member_create(const struct rte_member_parameters *params);
+
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Lookup key in set-summary (SS).
+ * Single key lookup and return as soon as the first match found
+ * @param setsum
+ * pointer of a setsummary
+ * @param key
+ * pointer of the key that needs to lookup
+ * @param set_id
+ * output the set id matches the key
+ * @return
+ * return 1 for found a match and 0 for not found a match
+ */
+
+int
+rte_member_lookup(const void *setsum,
+ const void *key, MEMBER_SET_TYPE *set_id);
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * lookup bulk of keys in set-summary (SS).
+ * Each key lookup returns as soon as the first match found
+ * @param setsum
+ * Pointer of a setsummary
+ * @param keys
+ * Pointer of bulk of keys that to be lookup
+ * @param num_keys
+ * Number of keys that will be lookup
+ * @param set_ids
+ * Output set ids for all the keys to this array.
+ * User should preallocate array that can contain all results, which size is
+ * the num_keys.
+ * @return
+ * The number of keys that found a match
+ */
+
+
+int
+rte_member_lookup_bulk(const void *setsum,
+ const void **keys, uint32_t num_keys,
+ MEMBER_SET_TYPE *set_ids);
+
+
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Lookup a key in set-summary (SS) for multiple matches.
+ * The key lookup will find all matched entries (multiple match).
+ * Note that for cache mode of HT, each key can have at most one match. This is
+ * because keys with same signature that maps to same bucket will overwrite
+ * each other. So multi-match lookup should be used for vBF and non-cache HT.
+ * @param setsum
+ * pointer of a set-summary
+ * @param key
+ * The key that to be lookup
+ * @param max_match_per_key
+ * User specified maximum number of matches for each key. The function returns
+ * as soon as this number of matches found for the key.
+ * @param set_id
+ * Output set ids for all the matches of the key. User needs to preallocate
+ * the array that can contain max_match_per_key number of results.
+ * @return
+ * The number of matches that found for the key.
+ * For cache mode HT set-summary, the number should be at most 1
+ */
+
+int
+rte_member_lookup_multi(const void *setsum,
+ const void *key, uint32_t max_match_per_key,
+ MEMBER_SET_TYPE *set_id);
+
+
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Lookup a bulk of keys in set-summary (SS) for multiple matches each key.
+ * Each key lookup will find all matched entries (multiple match).
+ * Note that for cache mode HT, each key can have at most one match. So
+ * multi-match function is mainly used for vBF and non-cache mode HT.
+ * @param setsum
+ * pointer of a setsummary
+ * @param keys
+ * The keys that to be lookup
+ * @param num_keys
+ * The number of keys that will be lookup
+ * @param max_match_per_key
+ * The possible maximum number of matches for each key
+ * @param match_count
+ * Output the number of matches for each key in an array
+ * @param set_ids
+ * Return set ids for all the matches of all keys. User pass in a preallocated
+ * 2D array with first dimension as key index and second dimension as match
+ * index. For example set_ids[bulk_size][max_match_per_key]
+ * @return
+ * The number of keys that found one or more matches in the set-summary
+ */
+
+int
+rte_member_lookup_multi_bulk(const void *setsum,
+ const void **keys, uint32_t num_keys,
+ uint32_t max_match_per_key,
+ uint32_t *match_count,
+ MEMBER_SET_TYPE *set_ids);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Insert key into set-summary (SS).
+ *
+ * @param setsum
+ * pointer of a set-summary
+ * @param key
+ * the key that needs to be added
+ * @param set_id
+ * The set id associated with the key that needs to be added. Different mode
+ * supports different set_id ranges. 0 cannot be used as set_id since
+ * RTE_MEMBER_NO_MATCH by default is set as 0.
+ * For HT mode, the set_id has range as [1, 0x7FFF], MSB is reserved.
+ * For vBF mode the set id is limited by the num_set parameter when create
+ * the set-summary.
+ * @return
+ * HT (cache mode) and vBF should never fail unless the set_id is not in the
+ * valid range. In such case -EINVAL is returned.
+ * For HT (non-cache mode) it could fail with -ENOSPC error code when table is
+ * full.
+ * For success it returns different values for different modes to provide
+ * extra information for users.
+ * Return 0 for HT (cache mode) if the add does not cause
+ * eviction, return 1 otherwise. Return 0 for HT mode if success, -ENOSPC for
+ * full, and 1 if cuckoo eviction happens. Always return 0 for vBF mode.
+ */
+
+int
+rte_member_add(const void *setsum, const void *key,
+ MEMBER_SET_TYPE set_id);
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * De-allocate memory used by set-summary.
+ * @param setsum
+ * Pointer to the set summary
+ */
+void
+rte_member_free(void *setsum);
+
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Reset the set-summary tables. E.g. reset bits to be 0 in BF,
+ * reset set_id in each entry to be RTE_MEMBER_NO_MATCH in HT based SS.
+ * @param setsum
+ * Pointer to the set-summary
+ */
+void
+rte_member_reset(const void *setsum);
+
+
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Delete items from the set-summary. Note that vBF does not support deletion
+ * in current implementation. For vBF, error code of -EINVAL will be returned.
+ * @param setsum
+ * Pointer to the set-summary
+ * @param key
+ * The key to be deleted
+ * @param set_id
+ * For HT mode, we need both key and its corresponding set_id to
+ * properly delete the key. Without set_id, we may delete other keys with the
+ * same signature
+ * @return
+ * If no entry found to delete, an error code of -ENOENT could be returned
+ */
+
+int
+rte_member_delete(void *setsum, const void *key,
+ MEMBER_SET_TYPE set_id);
+
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_MEMBER_H_ */
diff --git a/lib/librte_member/rte_member_version.map b/lib/librte_member/rte_member_version.map
new file mode 100644
index 0000000..b0977e1
--- /dev/null
+++ b/lib/librte_member/rte_member_version.map
@@ -0,0 +1,15 @@
+DPDK_17.11 {
+ global:
+
+ rte_member_create;
+ rte_member_lookup;
+ rte_member_lookup_bulk;
+ rte_member_lookup_multi;
+ rte_member_lookup_multi_bulk;
+ rte_member_add;
+ rte_member_free;
+ rte_member_reset;
+ rte_member_delete;
+
+ local: *;
+};
--
2.7.4
Stephen Hemminger
2017-08-22 03:59:05 UTC
Permalink
On Mon, 21 Aug 2017 17:19:47 -0700
Post by Yipeng Wang
+int
+rte_member_lookup(const void *ss, const void *key,
+ MEMBER_SET_TYPE *set_id)
+{
+ const struct rte_member_setsum *setsum = ss;
+ if (setsum == NULL || key == NULL || set_id == NULL)
+ return -EINVAL;
+
+ int ret = 0;
Don't needlessly initialize all variable.
In all code paths, ret is set.

In fact you could just make all cases of the switch into return
and compiler could optimize it as a tail call.
Luca Boccassi
2017-08-22 10:02:16 UTC
Permalink
Post by Yipeng Wang
Membership library is an extension and generalization of a
traditional
filter (for example Bloom Filter) structure. In general, the
Membership
library is a data structure that provides a "set-summary" and
responds
to set-membership queries of whether a certain element belongs to a
set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the
set-summary.
The results of the membership test is not 100% accurate. Certain
false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set-
summary"
is memory efficient and fast on lookup.
This patch add the main API definition.
---
 lib/Makefile                             |   2 +
 lib/librte_eal/common/eal_common_log.c   |   1 +
 lib/librte_eal/common/include/rte_log.h  |   1 +
 lib/librte_member/Makefile               |  48 +++
 lib/librte_member/rte_member.c           | 357 +++++++++++++++++++++
 lib/librte_member/rte_member.h           | 518
+++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  15 +
 7 files changed, 942 insertions(+)
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_version.map
diff --git a/lib/librte_member/Makefile b/lib/librte_member/Makefile
new file mode 100644
index 0000000..997c825
--- /dev/null
+++ b/lib/librte_member/Makefile
@@ -0,0 +1,48 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or
without
+#   modification, are permitted provided that the following
conditions
+#
+#     * Redistributions of source code must retain the above
copyright
+#       notice, this list of conditions and the following
disclaimer.
+#     * Redistributions in binary form must reproduce the above
copyright
+#       notice, this list of conditions and the following disclaimer
in
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of its
+#       contributors may be used to endorse or promote products
derived
+#       from this software without specific prior written
permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT
NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+
This breaks reproducibility as the output directory will be included
before the source directory, causing a race - please do something like:

CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3
Post by Yipeng Wang
+EXPORT_MAP := rte_member_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
--
Kind regards,
Luca Boccassi
Ferruh Yigit
2017-08-24 09:35:22 UTC
Permalink
Post by Luca Boccassi
Post by Yipeng Wang
Membership library is an extension and generalization of a
traditional
filter (for example Bloom Filter) structure. In general, the
Membership
library is a data structure that provides a "set-summary" and
responds
to set-membership queries of whether a certain element belongs to a
set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the
set-summary.
The results of the membership test is not 100% accurate. Certain
false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set-
summary"
is memory efficient and fast on lookup.
This patch add the main API definition.
---
 lib/Makefile                             |   2 +
 lib/librte_eal/common/eal_common_log.c   |   1 +
 lib/librte_eal/common/include/rte_log.h  |   1 +
 lib/librte_member/Makefile               |  48 +++
 lib/librte_member/rte_member.c           | 357 +++++++++++++++++++++
 lib/librte_member/rte_member.h           | 518
+++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  15 +
 7 files changed, 942 insertions(+)
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_version.map
diff --git a/lib/librte_member/Makefile b/lib/librte_member/Makefile
new file mode 100644
index 0000000..997c825
--- /dev/null
+++ b/lib/librte_member/Makefile
@@ -0,0 +1,48 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or
without
+#   modification, are permitted provided that the following
conditions
+#
+#     * Redistributions of source code must retain the above
copyright
+#       notice, this list of conditions and the following
disclaimer.
+#     * Redistributions in binary form must reproduce the above
copyright
+#       notice, this list of conditions and the following disclaimer
in
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of its
+#       contributors may be used to endorse or promote products
derived
+#       from this software without specific prior written
permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT
NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+
This breaks reproducibility as the output directory will be included
CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3
Can we remove "-I$(SRCDIR)" completely by first installing headers and
later compiling objects, all using $(RTE_OUT) only?

Do you think can this work?
Post by Luca Boccassi
Post by Yipeng Wang
+EXPORT_MAP := rte_member_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
Luca Boccassi
2017-08-24 09:55:44 UTC
Permalink
Post by Ferruh Yigit
Post by Luca Boccassi
Post by Yipeng Wang
Membership library is an extension and generalization of a
traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a
set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the
set-summary.
The results of the membership test is not 100% accurate. Certain
false positive or false negative probability could exist.
However,
comparing to a "full-blown" complete list of elements, a "set-
summary"
is memory efficient and fast on lookup.
This patch add the main API definition.
---
 lib/Makefile                             |   2 +
 lib/librte_eal/common/eal_common_log.c   |   1 +
 lib/librte_eal/common/include/rte_log.h  |   1 +
 lib/librte_member/Makefile               |  48 +++
 lib/librte_member/rte_member.c           | 357 +++++++++++++++++++++
 lib/librte_member/rte_member.h           | 518
+++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  15 +
 7 files changed, 942 insertions(+)
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_version.map
diff --git a/lib/librte_member/Makefile
b/lib/librte_member/Makefile
new file mode 100644
index 0000000..997c825
--- /dev/null
+++ b/lib/librte_member/Makefile
@@ -0,0 +1,48 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or
without
+#   modification, are permitted provided that the following
conditions
+#
+#     * Redistributions of source code must retain the above
copyright
+#       notice, this list of conditions and the following
disclaimer.
+#     * Redistributions in binary form must reproduce the above
copyright
+#       notice, this list of conditions and the following disclaimer
in
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of its
+#       contributors may be used to endorse or promote products
derived
+#       from this software without specific prior written
permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT
NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+
This breaks reproducibility as the output directory will be
included
CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3
Can we remove "-I$(SRCDIR)" completely by first installing headers and
later compiling objects, all using $(RTE_OUT) only?
Do you think can this work?
I'm not sure, it might - but given Bruce's effort to port to Meson I'm
not sure it's worth spending a lot of time doing big refactoring of the
existing build system
Post by Ferruh Yigit
Post by Luca Boccassi
Post by Yipeng Wang
+EXPORT_MAP := rte_member_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
--
Kind regards,
Luca Boccassi
Ferruh Yigit
2017-08-24 10:32:31 UTC
Permalink
Post by Luca Boccassi
Post by Ferruh Yigit
Post by Luca Boccassi
Post by Yipeng Wang
Membership library is an extension and generalization of a
traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a
set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the
set-summary.
The results of the membership test is not 100% accurate. Certain
false positive or false negative probability could exist.
However,
comparing to a "full-blown" complete list of elements, a "set-
summary"
is memory efficient and fast on lookup.
This patch add the main API definition.
---
 lib/Makefile                             |   2 +
 lib/librte_eal/common/eal_common_log.c   |   1 +
 lib/librte_eal/common/include/rte_log.h  |   1 +
 lib/librte_member/Makefile               |  48 +++
 lib/librte_member/rte_member.c           | 357
+++++++++++++++++++++
 lib/librte_member/rte_member.h           | 518
+++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  15 +
 7 files changed, 942 insertions(+)
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_version.map
diff --git a/lib/librte_member/Makefile
b/lib/librte_member/Makefile
new file mode 100644
index 0000000..997c825
--- /dev/null
+++ b/lib/librte_member/Makefile
@@ -0,0 +1,48 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or
without
+#   modification, are permitted provided that the following
conditions
+#
+#     * Redistributions of source code must retain the above
copyright
+#       notice, this list of conditions and the following
disclaimer.
+#     * Redistributions in binary form must reproduce the above
copyright
+#       notice, this list of conditions and the following disclaimer
in
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of its
+#       contributors may be used to endorse or promote products
derived
+#       from this software without specific prior written
permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT
NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+
This breaks reproducibility as the output directory will be
included
CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3
Can we remove "-I$(SRCDIR)" completely by first installing headers and
later compiling objects, all using $(RTE_OUT) only?
Do you think can this work?
I'm not sure, it might - but given Bruce's effort to port to Meson I'm
not sure it's worth spending a lot of time doing big refactoring of the
existing build system
Not big refactoring, following seems worked for me, if you would like to
test:

diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
index 13115d146..643da47da 100644
--- a/mk/rte.lib.mk
+++ b/mk/rte.lib.mk
@@ -59,14 +59,19 @@ endif


_BUILD = $(LIB)
-_INSTALL = $(INSTALL-FILES-y) $(SYMLINK-FILES-y) $(RTE_OUTPUT)/lib/$(LIB)
+PREINSTALL = $(SYMLINK-FILES-y)
+_INSTALL = $(INSTALL-FILES-y) $(RTE_OUTPUT)/lib/$(LIB)
_CLEAN = doclean

.PHONY: all
all: install

.PHONY: install
+ifeq ($(SYMLINK-FILES-y),)
install: build _postinstall
+else
+install: _preinstall build _postinstall
+endif

_postinstall: build
Post by Luca Boccassi
Post by Ferruh Yigit
Post by Luca Boccassi
Post by Yipeng Wang
+EXPORT_MAP := rte_member_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
Luca Boccassi
2017-09-02 12:55:23 UTC
Permalink
Post by Ferruh Yigit
Post by Luca Boccassi
Post by Ferruh Yigit
Post by Luca Boccassi
Post by Yipeng Wang
Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element
belongs to
a
set(s). A membership test for an element will return the set
this
element
belongs to or not-found if the element is never inserted into the
set-summary.
The results of the membership test is not 100% accurate. Certain
false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set-
summary"
is memory efficient and fast on lookup.
This patch add the main API definition.
---
 lib/Makefile                             |   2 +
 lib/librte_eal/common/eal_common_log.c   |   1 +
 lib/librte_eal/common/include/rte_log.h  |   1 +
 lib/librte_member/Makefile               |  48 +++
 lib/librte_member/rte_member.c           | 357
+++++++++++++++++++++
 lib/librte_member/rte_member.h           | 518
+++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  15 +
 7 files changed, 942 insertions(+)
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_version.map
diff --git a/lib/librte_member/Makefile
b/lib/librte_member/Makefile
new file mode 100644
index 0000000..997c825
--- /dev/null
+++ b/lib/librte_member/Makefile
@@ -0,0 +1,48 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or
without
+#   modification, are permitted provided that the following
conditions
+#
+#     * Redistributions of source code must retain the above
copyright
+#       notice, this list of conditions and the following
disclaimer.
+#     * Redistributions in binary form must reproduce the above
copyright
+#       notice, this list of conditions and the following
disclaimer
in
+#       the documentation and/or other materials provided
with
the
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names
of
its
+#       contributors may be used to endorse or promote products
derived
+#       from this software without specific prior written
permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING,
BUT
NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT
NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED
AND
ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY,
OR
TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT
OF
THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+
This breaks reproducibility as the output directory will be included
before the source directory, causing a race - please do
something
CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3
Can we remove "-I$(SRCDIR)" completely by first installing
headers
and
later compiling objects, all using $(RTE_OUT) only?
Do you think can this work?
I'm not sure, it might - but given Bruce's effort to port to Meson I'm
not sure it's worth spending a lot of time doing big refactoring of the
existing build system
Not big refactoring, following seems worked for me, if you would like to
diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
index 13115d146..643da47da 100644
--- a/mk/rte.lib.mk
+++ b/mk/rte.lib.mk
@@ -59,14 +59,19 @@ endif
 _BUILD = $(LIB)
-_INSTALL = $(INSTALL-FILES-y) $(SYMLINK-FILES-y)
$(RTE_OUTPUT)/lib/$(LIB)
+PREINSTALL = $(SYMLINK-FILES-y)
+_INSTALL = $(INSTALL-FILES-y) $(RTE_OUTPUT)/lib/$(LIB)
 _CLEAN = doclean
 .PHONY: all
 all: install
 .PHONY: install
+ifeq ($(SYMLINK-FILES-y),)
 install: build _postinstall
+else
+install: _preinstall build _postinstall
+endif
 _postinstall: build
Easier than I would have thought :-)

I'll give that patch a spin through the repro build grinder next week,
thanks.
--
Kind regards,
Luca Boccassi
Luca Boccassi
2017-09-02 23:49:34 UTC
Permalink
Post by Luca Boccassi
Post by Ferruh Yigit
Post by Luca Boccassi
Post by Ferruh Yigit
Post by Luca Boccassi
Post by Yipeng Wang
Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the
Membership
library is a data structure that provides a "set-summary" and
responds
to set-membership queries of whether a certain element
belongs to
a
set(s). A membership test for an element will return the set
this
element
belongs to or not-found if the element is never inserted
into
the
set-summary.
The results of the membership test is not 100% accurate. Certain
false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set-
summary"
is memory efficient and fast on lookup.
This patch add the main API definition.
---
 lib/Makefile                             |   2 +
 lib/librte_eal/common/eal_common_log.c   |   1 +
 lib/librte_eal/common/include/rte_log.h  |   1 +
 lib/librte_member/Makefile               |  48 +++
 lib/librte_member/rte_member.c           | 357
+++++++++++++++++++++
 lib/librte_member/rte_member.h           | 518
+++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  15 +
 7 files changed, 942 insertions(+)
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644
lib/librte_member/rte_member_version.map
diff --git a/lib/librte_member/Makefile
b/lib/librte_member/Makefile
new file mode 100644
index 0000000..997c825
--- /dev/null
+++ b/lib/librte_member/Makefile
@@ -0,0 +1,48 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms,
with
or
without
+#   modification, are permitted provided that the following
conditions
+#
+#     * Redistributions of source code must retain the above
copyright
+#       notice, this list of conditions and the following
disclaimer.
+#     * Redistributions in binary form must reproduce the above
copyright
+#       notice, this list of conditions and the following
disclaimer
in
+#       the documentation and/or other materials provided
with
the
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names
of
its
+#       contributors may be used to endorse or promote products
derived
+#       from this software without specific prior written
permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING,
BUT
NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING,
BUT
NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES;
LOSS OF
USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED
AND
ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY,
OR
TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT
OF
THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY
OF
SUCH
DAMAGE.
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+
This breaks reproducibility as the output directory will be included
before the source directory, causing a race - please do
something
CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3
Can we remove "-I$(SRCDIR)" completely by first installing
headers
and
later compiling objects, all using $(RTE_OUT) only?
Do you think can this work?
I'm not sure, it might - but given Bruce's effort to port to
Meson
I'm
not sure it's worth spending a lot of time doing big refactoring
of
the
existing build system
Not big refactoring, following seems worked for me, if you would
like
to
diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
index 13115d146..643da47da 100644
--- a/mk/rte.lib.mk
+++ b/mk/rte.lib.mk
@@ -59,14 +59,19 @@ endif
 _BUILD = $(LIB)
-_INSTALL = $(INSTALL-FILES-y) $(SYMLINK-FILES-y)
$(RTE_OUTPUT)/lib/$(LIB)
+PREINSTALL = $(SYMLINK-FILES-y)
+_INSTALL = $(INSTALL-FILES-y) $(RTE_OUTPUT)/lib/$(LIB)
 _CLEAN = doclean
 .PHONY: all
 all: install
 .PHONY: install
+ifeq ($(SYMLINK-FILES-y),)
 install: build _postinstall
+else
+install: _preinstall build _postinstall
+endif
 _postinstall: build
Easier than I would have thought :-)
I'll give that patch a spin through the repro build grinder next week,
thanks.
Thanks Ferruh, this works in making the build reproducible with the
current CI as an alternative with a fixed build path.

But on the other hand, it will cause the compiler to always embed the
custom build directory in the path to the header in the DWARF .debug
files, rather than the "repository" path:

readelf --debug-dump=line /tmp/69c02db5050024b530a14555b49528c419419f.debug

<cut>
 The Directory Table (offset 0x1b):
  1     dpdk/lib/librte_eal/linuxapp/eal
  2     /usr/include/x86_64-linux-gnu/bits
  3     dpdk/lib/librte_eal/common
  4     dpdk/debian/build/shared-root/include
  5     dpdk/debian/build/shared-root/include/generic
  6     /usr/lib/gcc/x86_64-linux-gnu/7/include
  7     /usr/include/x86_64-linux-gnu/sys
  8     /usr/include

 The File Name Table (offset 0x15a):
  Entry Dir     Time    Size    Name
  1     1       0       0       eal.c
  2     2       0       0       stdio2.h
  3     3       0       0       eal_filesystem.h
  4     2       0       0       fcntl2.h
  5     4       0       0       rte_atomic.h
  6     5       0       0       rte_atomic.h
  7     2       0       0       string3.h
  8     4       0       0       rte_cycles.h
  9     4       0       0       rte_random.h
  10    6       0       0       xmmintrin.h
  11    4       0       0       rte_eal_memconfig.h
  12    4       0       0       rte_lcore.h

<cut>

note 4 and 5 items in the directory tables, they point to debian/build-
shared-root which is the output directory we use in the package build.

This means that 2 builds that are otherwise exactly the same (same
machine type, same build dependencies) will not be binary identical due
to that difference.

On one hand it's a much smaller and nicer patch and there's the Meson
work which will make both obsolete, on the other hand it's not very
reproducibl-y. What do you think?
--
Kind regards,
Luca Boccassi
Wang, Yipeng1
2017-08-24 18:38:11 UTC
Permalink
-----Original Message-----
Sent: Tuesday, August 22, 2017 3:02 AM
Subject: Re: [dpdk-dev] [PATCH 1/7] member: implement main API
Post by Yipeng Wang
Membership library is an extension and generalization of a
traditional
filter (for example Bloom Filter) structure. In general, the
Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a
set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the
set-summary.
The results of the membership test is not 100% accurate. Certain
false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set-
summary"
is memory efficient and fast on lookup.
This patch add the main API definition.
---
 lib/Makefile                             |   2 +
 lib/librte_eal/common/eal_common_log.c   |   1 +
 lib/librte_eal/common/include/rte_log.h  |   1 +
 lib/librte_member/Makefile               |  48 +++
 lib/librte_member/rte_member.c           | 357 +++++++++++++++++++++
 lib/librte_member/rte_member.h           | 518
+++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  15 +
 7 files changed, 942 insertions(+)
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_version.map
diff --git a/lib/librte_member/Makefile b/lib/librte_member/Makefile
new file mode 100644
index 0000000..997c825
--- /dev/null
+++ b/lib/librte_member/Makefile
@@ -0,0 +1,48 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or
without
+#   modification, are permitted provided that the following
conditions
+#
+#     * Redistributions of source code must retain the above
copyright
+#       notice, this list of conditions and the following
disclaimer.
+#     * Redistributions in binary form must reproduce the above
copyright
+#       notice, this list of conditions and the following disclaimer
in
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of its
+#       contributors may be used to endorse or promote products
derived
+#       from this software without specific prior written
permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT
NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT NOT
Post by Yipeng Wang
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF
Post by Yipeng Wang
USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
AND
Post by Yipeng Wang
ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
OF
Post by Yipeng Wang
THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+
This breaks reproducibility as the output directory will be included
CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3
[Wang, Yipeng]
Thanks Luca, I will reverse the include order in next version.

I am not an expert on the build system, to be more clear, the race condition you
mentioned is because the library may search build directory first that some symlinks
may have not been created yet, right?

Thanks
Post by Yipeng Wang
+EXPORT_MAP := rte_member_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
--
Kind reg
Luca Boccassi
2017-09-02 12:54:38 UTC
Permalink
-----Original Message-----
Sent: Tuesday, August 22, 2017 3:02 AM
Subject: Re: [dpdk-dev] [PATCH 1/7] member: implement main API
Post by Yipeng Wang
Membership library is an extension and generalization of a
traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a
set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the
set-summary.
The results of the membership test is not 100% accurate. Certain
false positive or false negative probability could exist.
However,
comparing to a "full-blown" complete list of elements, a "set-
summary"
is memory efficient and fast on lookup.
This patch add the main API definition.
---
 lib/Makefile                             |   2 +
 lib/librte_eal/common/eal_common_log.c   |   1 +
 lib/librte_eal/common/include/rte_log.h  |   1 +
 lib/librte_member/Makefile               |  48 +++
 lib/librte_member/rte_member.c           | 357 +++++++++++++++++++++
 lib/librte_member/rte_member.h           | 518
+++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  15 +
 7 files changed, 942 insertions(+)
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_version.map
diff --git a/lib/librte_member/Makefile
b/lib/librte_member/Makefile
new file mode 100644
index 0000000..997c825
--- /dev/null
+++ b/lib/librte_member/Makefile
@@ -0,0 +1,48 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or
without
+#   modification, are permitted provided that the following
conditions
+#
+#     * Redistributions of source code must retain the above
copyright
+#       notice, this list of conditions and the following
disclaimer.
+#     * Redistributions in binary form must reproduce the above
copyright
+#       notice, this list of conditions and the following disclaimer
in
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of its
+#       contributors may be used to endorse or promote products
derived
+#       from this software without specific prior written
permission.
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT
NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT NOT
Post by Yipeng Wang
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF
Post by Yipeng Wang
USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
AND
Post by Yipeng Wang
ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
OF
Post by Yipeng Wang
THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+
This breaks reproducibility as the output directory will be
included
CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3
[Wang, Yipeng] 
Thanks Luca, I will reverse the include order in next version.
I am not an expert on the build system, to be more clear, the race
condition you 
mentioned is because the library may search build directory first that some symlinks
may have not been created yet, right?
Thanks
Yes, and as a consequence on some builds the DWARF symbols list
lib/libfoo/foo.h as the source file path in the .debug, and on some
builds they list build-dir/include/foo.h
Post by Yipeng Wang
+EXPORT_MAP := rte_member_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
--
Kind regards,
Luca Boccassi
--
Kind regards,
Luca Boccassi
Yipeng Wang
2017-08-22 00:19:48 UTC
Permalink
One of the set-summray structure is hash-table based
set-summary (HTSS). One example is cuckoo filter [1].

Comparing to a traditional hash table, HTSS has a much more
compact structure. For each element, only one signature and
its corresponding set ID is stored. No key comparison is required
during lookup. For the table structure, there are multiple entries
in each bucket, and the table is composed of many buckets.

Two modes are supported for HTSS, "cache" and "none-cache" modes.
The non-cache mode is more similar to traditional cuckoo filter.
When a bucket is full, one entry will be evicted to its
alternative bucket to make space for the new key. The table could
be full and then no more keys could be inserted. This mode has
false-positive rate but no false-negative. Multiple entries
with same signature could stay in the same bucket.

The "cache" mode does not evict key to its alternative bucket
when a bucket is full, an existing key will be evicted out of
the table like a cache. So the table will never reject keys when
it is full. Another property is in each bucket, there cannot be
multiple entries with same signature. The mode could have both
false-positive and false-negative probability.

This patch add the implementation of HTSS.

[1] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically
Better Than Bloom,” in Conference on emerging Networking
Experiments and Technologies, 2014.

Signed-off-by: Yipeng Wang <***@intel.com>
---
lib/librte_member/Makefile | 2 +-
lib/librte_member/rte_member_ht.c | 488 ++++++++++++++++++++++++++++++++++++++
lib/librte_member/rte_member_ht.h | 115 +++++++++
3 files changed, 604 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_member/rte_member_ht.c
create mode 100644 lib/librte_member/rte_member_ht.h

diff --git a/lib/librte_member/Makefile b/lib/librte_member/Makefile
index 997c825..110e4be 100644
--- a/lib/librte_member/Makefile
+++ b/lib/librte_member/Makefile
@@ -41,7 +41,7 @@ EXPORT_MAP := rte_member_version.map
LIBABIVER := 1

# all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c
# install includes
SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h

diff --git a/lib/librte_member/rte_member_ht.c b/lib/librte_member/rte_member_ht.c
new file mode 100644
index 0000000..3e411ed
--- /dev/null
+++ b/lib/librte_member/rte_member_ht.c
@@ -0,0 +1,488 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <rte_errno.h>
+#include <rte_malloc.h>
+#include <rte_prefetch.h>
+#include <rte_random.h>
+#include <rte_log.h>
+
+#include "rte_member.h"
+#include "rte_member_ht.h"
+
+
+static inline int
+insert_overwrite_search(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ MEMBER_SET_TYPE set_id)
+{
+ int i;
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ if (buckets[bucket].sigs[i] == tmp_sig) {
+ buckets[bucket].sets[i] = set_id;
+ return 1;
+ }
+ }
+ return 0;
+}
+
+
+
+
+static inline int
+search_bucket_single(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ MEMBER_SET_TYPE *set_id)
+{
+ int iter;
+ for (iter = 0; iter < RTE_MEMBER_BUCKET_ENTRIES; iter++) {
+ if (tmp_sig == buckets[bucket].sigs[iter] &&
+ buckets[bucket].sets[iter] !=
+ RTE_MEMBER_NO_MATCH) {
+ *set_id = buckets[bucket].sets[iter];
+ return 1;
+ }
+ }
+ return 0;
+}
+
+
+static inline void
+search_bucket_multi(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ uint32_t *counter,
+ uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id)
+{
+ int iter;
+ for (iter = 0; iter < RTE_MEMBER_BUCKET_ENTRIES; iter++) {
+ if (tmp_sig == buckets[bucket].sigs[iter] &&
+ buckets[bucket].sets[iter] !=
+ RTE_MEMBER_NO_MATCH) {
+ set_id[*counter] = buckets[bucket].sets[iter];
+ (*counter)++;
+ if (*counter >= match_per_key)
+ return;
+ }
+ }
+}
+
+
+int
+rte_member_create_ht(struct rte_member_setsum *ss,
+ const struct rte_member_parameters *params)
+{
+ uint32_t i, j;
+ uint32_t size_bucket_t;
+
+ const uint32_t num_buckets = rte_align32pow2(params->num_keys) /
+ RTE_MEMBER_BUCKET_ENTRIES;
+
+ if ((num_buckets / RTE_MEMBER_BUCKET_ENTRIES >
+ RTE_MEMBER_ENTRIES_MAX) ||
+ !rte_is_power_of_2(RTE_MEMBER_BUCKET_ENTRIES)) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, MEMBER,
+ "Membership HT create with invalid parameters\n");
+ }
+ size_bucket_t = sizeof(struct member_ht_bucket);
+
+ struct member_ht_bucket *buckets = rte_zmalloc_socket(NULL,
+ num_buckets * size_bucket_t,
+ RTE_CACHE_LINE_SIZE, ss->socket_id);
+
+ if (buckets == NULL)
+ return -ENOMEM;
+
+ ss->table = buckets;
+ ss->bucket_cnt = num_buckets;
+ ss->bucket_mask = num_buckets - 1;
+ ss->cache = params->iscache;
+
+ for (i = 0; i < num_buckets; i++) {
+ for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++)
+ buckets[i].sets[j] = RTE_MEMBER_NO_MATCH;
+ }
+
+
+ RTE_LOG(DEBUG, MEMBER, "Hash table based filter created, "
+ "the table has %u entries, %u buckets\n",
+ num_buckets,
+ num_buckets / RTE_MEMBER_BUCKET_ENTRIES);
+ return 0;
+}
+
+static inline
+void get_buckets_index(const struct rte_member_setsum *ss, const void *key,
+ uint32_t *prim_bkt, uint32_t *sec_bkt, SIG_TYPE *sig)
+{
+ uint32_t first_hash = MEMBER_PRIM_HASH(key, ss->key_len,
+ ss->prim_hash_seed);
+ uint32_t sec_hash = MEMBER_SEC_HASH(&first_hash, 4, ss->sec_hash_seed);
+ *sig = first_hash & SIG_BITMASK;
+ if (ss->cache) {
+ *prim_bkt = sec_hash & ss->bucket_mask;
+ *sec_bkt = (sec_hash >> 16) & ss->bucket_mask;
+ } else {
+ *prim_bkt = sec_hash & ss->bucket_mask;
+ *sec_bkt = (*prim_bkt ^ *sig) & ss->bucket_mask;
+ }
+}
+
+
+int
+rte_member_lookup_ht(const struct rte_member_setsum *ss,
+ const void *key, MEMBER_SET_TYPE *set_id)
+{
+ uint32_t prim_bucket, sec_bucket;
+ SIG_TYPE tmp_sig;
+ struct member_ht_bucket *buckets = ss->table;
+
+
+ *set_id = RTE_MEMBER_NO_MATCH;
+ get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
+
+ if (search_bucket_single(prim_bucket, tmp_sig, buckets,
+ set_id) ||
+ search_bucket_single(sec_bucket, tmp_sig,
+ buckets, set_id))
+ return 1;
+
+ return 0;
+}
+
+
+uint32_t
+rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss,
+ const void **keys, uint32_t num_keys, MEMBER_SET_TYPE *set_id)
+{
+ uint32_t i;
+ uint32_t ret = 0;
+ struct member_ht_bucket *buckets = ss->table;
+ SIG_TYPE tmp_sig[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+ uint32_t prim_buckets[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+ uint32_t sec_buckets[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+
+ for (i = 0; i < num_keys; i++) {
+ get_buckets_index(ss, keys[i], &prim_buckets[i],
+ &sec_buckets[i], &tmp_sig[i]);
+ rte_prefetch0(&buckets[prim_buckets[i]]);
+ rte_prefetch0(&buckets[sec_buckets[i]]);
+ }
+
+ for (i = 0; i < num_keys; i++) {
+ if (search_bucket_single(prim_buckets[i], tmp_sig[i],
+ buckets, &set_id[i]) ||
+ search_bucket_single(sec_buckets[i],
+ tmp_sig[i], buckets, &set_id[i]))
+ ret++;
+ else
+ set_id[i] = RTE_MEMBER_NO_MATCH;
+ }
+ return ret;
+}
+
+
+uint32_t
+rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,
+ const void *key, uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id)
+{
+ uint32_t ret = 0;
+ uint32_t prim_bucket, sec_bucket;
+ SIG_TYPE tmp_sig;
+ struct member_ht_bucket *buckets = ss->table;
+
+ get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
+
+ search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
+ match_per_key, set_id);
+ if (ret < match_per_key)
+ search_bucket_multi(sec_bucket, tmp_sig,
+ buckets, &ret, match_per_key, set_id);
+ return ret;
+}
+
+
+uint32_t
+rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,
+ const void **keys, uint32_t num_keys, uint32_t match_per_key,
+ uint32_t *match_count,
+ MEMBER_SET_TYPE *set_ids)
+{
+ uint32_t i;
+ uint32_t ret = 0;
+ struct member_ht_bucket *buckets = ss->table;
+ uint32_t match_cnt_t;
+ SIG_TYPE tmp_sig[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+ uint32_t prim_buckets[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+ uint32_t sec_buckets[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+
+ for (i = 0; i < num_keys; i++) {
+ get_buckets_index(ss, keys[i], &prim_buckets[i],
+ &sec_buckets[i], &tmp_sig[i]);
+ rte_prefetch0(&buckets[prim_buckets[i]]);
+ rte_prefetch0(&buckets[sec_buckets[i]]);
+ }
+ for (i = 0; i < num_keys; i++) {
+ match_cnt_t = 0;
+
+ search_bucket_multi(prim_buckets[i], tmp_sig[i],
+ buckets, &match_cnt_t, match_per_key,
+ &set_ids[i*match_per_key]);
+ if (match_cnt_t < match_per_key)
+ search_bucket_multi(sec_buckets[i], tmp_sig[i],
+ buckets, &match_cnt_t, match_per_key,
+ &set_ids[i*match_per_key]);
+ match_count[i] = match_cnt_t;
+ if (match_cnt_t != 0)
+ ret++;
+ }
+ return ret;
+}
+
+static inline int
+try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
+ SIG_TYPE sig, MEMBER_SET_TYPE set_id)
+{
+ int i;
+ /* If not full then insert into one slot*/
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ if (buckets[prim].sets[i] == RTE_MEMBER_NO_MATCH) {
+ buckets[prim].sigs[i] = sig;
+ buckets[prim].sets[i] = set_id;
+ return 0;
+ }
+ }
+ /* if prim failed, we need to access second cache line */
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ if (buckets[sec].sets[i] == RTE_MEMBER_NO_MATCH) {
+ buckets[sec].sigs[i] = sig;
+ buckets[sec].sets[i] = set_id;
+ return 0;
+ }
+ }
+ return -1;
+}
+
+
+static inline int
+try_overwrite(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
+ SIG_TYPE sig, MEMBER_SET_TYPE set_id)
+{
+ if (insert_overwrite_search(prim, sig, buckets, set_id) ||
+ insert_overwrite_search(sec, sig, buckets,
+ set_id))
+ return 0;
+ return -1;
+}
+
+static inline int
+evict_from_bucket(void)
+{
+ /* for now, we randomly pick one entry to evict */
+ return rte_rand() & (RTE_MEMBER_BUCKET_ENTRIES - 1);
+}
+
+
+
+/*
+ * This function is similar to the cuckoo hash make_space function in hash
+ * library
+ */
+static inline int
+make_space_bucket(const struct rte_member_setsum *ss, uint32_t bkt_num)
+{
+
+ static unsigned int nr_pushes;
+
+ unsigned int i, j;
+ int ret;
+ struct member_ht_bucket *buckets = ss->table;
+ uint32_t next_bucket_idx;
+ struct member_ht_bucket *next_bkt[RTE_MEMBER_BUCKET_ENTRIES];
+ struct member_ht_bucket *bkt = &buckets[bkt_num];
+ MEMBER_SET_TYPE flag_mask = 1U << (sizeof(MEMBER_SET_TYPE) * 8 - 1);
+ /*
+ * Push existing item (search for bucket with space in
+ * alternative locations) to its alternative location
+ */
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ /* Search for space in alternative locations */
+ next_bucket_idx = (bkt->sigs[i] ^ bkt_num) & ss->bucket_mask;
+ next_bkt[i] = &buckets[next_bucket_idx];
+ for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++) {
+ if (next_bkt[i]->sets[j] == RTE_MEMBER_NO_MATCH)
+ break;
+ }
+
+ if (j != RTE_MEMBER_BUCKET_ENTRIES)
+ break;
+ }
+
+ /* Alternative location has spare room (end of recursive function) */
+ if (i != RTE_MEMBER_BUCKET_ENTRIES) {
+ next_bkt[i]->sigs[j] = bkt->sigs[i];
+ next_bkt[i]->sets[j] = bkt->sets[i];
+ return i;
+ }
+
+ /* Pick entry that has not been pushed yet */
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++)
+ if ((bkt->sets[i] & flag_mask) == 0)
+ break;
+
+ /* All entries have been pushed, so entry cannot be added */
+ if (i == RTE_MEMBER_BUCKET_ENTRIES ||
+ nr_pushes > RTE_MEMBER_MAX_PUSHES)
+ return -ENOSPC;
+
+ next_bucket_idx = (bkt->sigs[i] ^ bkt_num) & ss->bucket_mask;
+ /* Set flag to indicate that this entry is going to be pushed */
+ bkt->sets[i] |= flag_mask;
+
+ nr_pushes++;
+ /* Need room in alternative bucket to insert the pushed entry */
+ ret = make_space_bucket(ss, next_bucket_idx);
+ /*
+ * After recursive function.
+ * Clear flags and insert the pushed entry
+ * in its alternative location if successful,
+ * or return error
+ */
+ bkt->sets[i] &= ~flag_mask;
+ nr_pushes = 0;
+ if (ret >= 0) {
+ next_bkt[i]->sigs[ret] = bkt->sigs[i];
+ next_bkt[i]->sets[ret] = bkt->sets[i];
+ return i;
+ } else
+ return ret;
+}
+
+
+int
+rte_member_add_ht(const struct rte_member_setsum *ss,
+ const void *key, MEMBER_SET_TYPE set_id)
+{
+ int ret;
+ uint32_t prim_bucket, sec_bucket;
+ SIG_TYPE tmp_sig;
+ struct member_ht_bucket *buckets = ss->table;
+ MEMBER_SET_TYPE flag_mask = 1U << (sizeof(MEMBER_SET_TYPE) * 8 - 1);
+
+ if (set_id == RTE_MEMBER_NO_MATCH || (set_id & flag_mask) != 0)
+ return -EINVAL;
+
+ get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
+
+ /* if it is cache based filter, we try overwriting existing entry */
+ if (ss->cache) {
+ ret = try_overwrite(buckets, prim_bucket, sec_bucket, tmp_sig,
+ set_id);
+ if (ret != -1)
+ return ret;
+ }
+ /* If not full then insert into one slot*/
+ ret = try_insert(buckets, prim_bucket, sec_bucket, tmp_sig, set_id);
+ if (ret != -1)
+ return ret;
+
+ /* random pick prim or sec for recursive displacement */
+
+ uint32_t select_bucket = (tmp_sig && 1U) ? prim_bucket : sec_bucket;
+ if (ss->cache) {
+ ret = evict_from_bucket();
+ buckets[select_bucket].sigs[ret] = tmp_sig;
+ buckets[select_bucket].sets[ret] = set_id;
+ return 1;
+ }
+
+ ret = make_space_bucket(ss, select_bucket);
+ if (ret >= 0) {
+ buckets[select_bucket].sigs[ret] = tmp_sig;
+ buckets[select_bucket].sets[ret] = set_id;
+ ret = 1;
+ }
+
+ return ret;
+}
+
+
+void
+rte_member_free_ht(struct rte_member_setsum *ss)
+{
+ rte_free(ss->table);
+}
+
+
+int
+rte_member_delete_ht(struct rte_member_setsum *ss, const void *key,
+ MEMBER_SET_TYPE set_id)
+{
+ int i;
+ uint32_t prim_bucket, sec_bucket;
+ SIG_TYPE tmp_sig;
+ struct member_ht_bucket *buckets = ss->table;
+
+ get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
+
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ if (tmp_sig == buckets[prim_bucket].sigs[i] &&
+ set_id == buckets[prim_bucket].sets[i]) {
+ buckets[prim_bucket].sets[i] = RTE_MEMBER_NO_MATCH;
+ return 0;
+ }
+ }
+
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ if (tmp_sig == buckets[sec_bucket].sigs[i] &&
+ set_id == buckets[sec_bucket].sets[i]) {
+ buckets[sec_bucket].sets[i] = RTE_MEMBER_NO_MATCH;
+ return 0;
+ }
+ }
+ return -ENOENT;
+}
+
+
+void
+rte_member_reset_ht(const struct rte_member_setsum *ss)
+{
+ uint32_t i, j;
+ struct member_ht_bucket *buckets = ss->table;
+ for (i = 0; i < ss->bucket_cnt; i++) {
+ for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++)
+ buckets[i].sets[j] = RTE_MEMBER_NO_MATCH;
+ }
+}
diff --git a/lib/librte_member/rte_member_ht.h b/lib/librte_member/rte_member_ht.h
new file mode 100644
index 0000000..aebf1db
--- /dev/null
+++ b/lib/librte_member/rte_member_ht.h
@@ -0,0 +1,115 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _RTE_MEMBER_HT_H_
+#define _RTE_MEMBER_HT_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* Maximum number of pushes for cuckoo path in HT mode. */
+#define RTE_MEMBER_MAX_PUSHES 100
+
+typedef uint16_t SIG_TYPE; /* signature size is 16 bit */
+#define SIG_BITMASK 0xFFFF /* signature mask, 16 bit */
+
+
+/* The bucket struct for ht setsum */
+struct member_ht_bucket {
+ SIG_TYPE sigs[RTE_MEMBER_BUCKET_ENTRIES]; /* 2-byte signature */
+ MEMBER_SET_TYPE sets[RTE_MEMBER_BUCKET_ENTRIES]; /* 2-byte set */
+} __rte_cache_aligned;
+
+
+
+int
+rte_member_create_ht(struct rte_member_setsum *ss,
+ const struct rte_member_parameters *params);
+
+int
+rte_member_lookup_ht(const struct rte_member_setsum *setsum,
+ const void *key, MEMBER_SET_TYPE *set_id);
+
+
+
+uint32_t
+rte_member_lookup_bulk_ht(const struct rte_member_setsum *setsum,
+ const void **keys, uint32_t num_keys,
+ MEMBER_SET_TYPE *set_ids);
+
+
+
+
+uint32_t
+rte_member_lookup_multi_ht(const struct rte_member_setsum *setsum,
+ const void *key, uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id);
+
+
+
+uint32_t
+rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *setsum,
+ const void **keys, uint32_t num_keys, uint32_t match_per_key,
+ uint32_t *match_count,
+ MEMBER_SET_TYPE *set_ids);
+
+
+
+
+int
+rte_member_add_ht(const struct rte_member_setsum *setsum,
+ const void *key, MEMBER_SET_TYPE set_id);
+
+void
+rte_member_free_ht(struct rte_member_setsum *setsum);
+
+
+
+
+int
+rte_member_delete_ht(struct rte_member_setsum *ss, const void *key,
+ MEMBER_SET_TYPE set_id);
+
+
+void
+rte_member_reset_ht(const struct rte_member_setsum *setsum);
+
+
+#ifdef __cplusplus
+}
+#endif
+
+
+
+#endif /* _RTE_MEMBER_HT_H_ */
--
2.7.4
Yipeng Wang
2017-08-22 00:19:49 UTC
Permalink
Bloom Filter (BF) [1] is a well-known space-efficient
probabilistic data structure that answers set membership queries.
Vector of Bloom Filters (vBF) is an extension to traditional BF
that supports multi-set membership testing. Traditional BF will
return found or not-found for each key. vBF will also return
which set the key belongs to if it is found.

Since each set requires a BF, vBF should be used when set count
is small. vBF's false positive rate could be set arbitrarily so
that its memory requirement and lookup speed is better in certain
cases comparing to HT based set-summary.

This patch adds the vBF implementation.

[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable
Errors,” Communications of the ACM, 1970.

Signed-off-by: Yipeng Wang <***@intel.com>
---
lib/librte_member/Makefile | 2 +-
lib/librte_member/rte_member_vbf.c | 309 +++++++++++++++++++++++++++++++++++++
lib/librte_member/rte_member_vbf.h | 85 ++++++++++
3 files changed, 395 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_member/rte_member_vbf.c
create mode 100644 lib/librte_member/rte_member_vbf.h

diff --git a/lib/librte_member/Makefile b/lib/librte_member/Makefile
index 110e4be..90bdfe8 100644
--- a/lib/librte_member/Makefile
+++ b/lib/librte_member/Makefile
@@ -41,7 +41,7 @@ EXPORT_MAP := rte_member_version.map
LIBABIVER := 1

# all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c rte_member_vbf.c
# install includes
SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h

diff --git a/lib/librte_member/rte_member_vbf.c b/lib/librte_member/rte_member_vbf.c
new file mode 100644
index 0000000..90c10ea
--- /dev/null
+++ b/lib/librte_member/rte_member_vbf.c
@@ -0,0 +1,309 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <math.h>
+#include <string.h>
+
+#include <rte_malloc.h>
+#include <rte_memory.h>
+#include <rte_errno.h>
+#include <rte_log.h>
+
+#include "rte_member.h"
+#include "rte_member_vbf.h"
+
+/*
+ * vBF currently implemented as a big array.
+ * The BFs have a vertical layout. Bits in same location of all bfs will stay
+ * in the same cache line.
+ * For example, if we have 32 bloom filters, we use a uint32_t array to
+ * represent all of them. array[0] represent the first location of all the
+ * bloom filters, array[1] represents the second location of all the
+ * bloom filters, etc. The advantage of this layout is to minimize the average
+ * number of memory accesses to test all bloom filters.
+ *
+ * Currently the implementation supports vBF containing 1,2,4,8,16,32 BFs.
+ */
+
+int
+rte_member_create_vbf(struct rte_member_setsum *ss,
+ const struct rte_member_parameters *params)
+{
+
+ if (params->num_set > 32 || !rte_is_power_of_2(params->num_set)) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, MEMBER, "vBF create with invalid parameters\n");
+ }
+
+ /* We assume expected keys evenly distribute to all BFs */
+ uint32_t num_keys_per_bf = (params->num_keys + ss->num_set - 1) /
+ ss->num_set;
+
+
+ /*
+ * Note that the false positive rate is for all BFs in the vBF
+ * such that the single BF's false positive rate need to be
+ * calculated.
+ * Assume each BF's False positive rate is x. The total false positive
+ * rate is fp = 1-(1-x)^n.
+ * => x = 1 - (1-fp)^(1/n)
+ */
+
+ float x = 1 - pow((1 - params->false_positive_rate), 1.0 / ss->num_set);
+
+ uint32_t bits = ceil((num_keys_per_bf *
+ log(x)) / log(1.0 / (pow(2.0, log(2.0)))));
+
+ /* We round to power of 2 for performance during lookup */
+ ss->bits = rte_align32pow2(bits);
+
+ ss->num_hashes = (uint32_t)(log(2.0) * bits / num_keys_per_bf);
+ ss->bit_mask = ss->bits - 1;
+
+ ss->table = rte_zmalloc_socket(NULL, sizeof(uint32_t) * ss->num_set *
+ (ss->bits >> 5), RTE_CACHE_LINE_SIZE, ss->socket_id);
+
+
+ /*
+ * Since we round the bits to power of 2, the final false positive
+ * rate will probably not be same as the user specified. We log the
+ * new value as debug message.
+ */
+ float new_fp = pow((1 - pow((1 - 1.0 / ss->bits), num_keys_per_bf *
+ ss->num_hashes)), ss->num_hashes);
+ new_fp = 1 - pow((1 - new_fp), ss->num_set);
+
+ RTE_LOG(DEBUG, MEMBER, "vector bloom filter created, "
+ "each bloom filter expects %u keys, needs %u bits, %u hashes, "
+ "with false positive rate set as %.5f, "
+ "The new calculated vBF false positive rate is %.5f\n",
+ num_keys_per_bf, ss->bits, ss->num_hashes, x, new_fp);
+
+ if (ss->table == NULL)
+ return -ENOMEM;
+
+ return 0;
+}
+
+
+/*
+ * a is how many bits could be represented by one uint32_t variable.
+ * b is used for division shift
+ * shift is used for multiplication shift
+ */
+static inline uint32_t
+test_bit(uint32_t h1, uint32_t h2, uint32_t iter, uint32_t shift, uint32_t a,
+ uint32_t b, const struct rte_member_setsum *ss)
+{
+ uint32_t *vbf = ss->table;
+ uint32_t n = ss->num_set;
+ uint32_t bit_loc = (h1 + iter * h2) & ss->bit_mask;
+ /*
+ * x>>b is the divide, x & (a-1) is the mod, & (1<<n-1) to mask out bits
+ * we do not need
+ */
+ return (vbf[bit_loc>>b] >> ((bit_loc & (a - 1)) << shift)) &
+ ((1ULL << n) - 1);
+}
+
+
+static inline void
+set_bit(uint32_t h1, uint32_t h2, uint32_t iter, uint32_t shift, uint32_t a,
+ uint32_t b, const struct rte_member_setsum *ss, int32_t set)
+{
+ uint32_t *vbf = ss->table;
+ uint32_t bit_loc = (h1 + iter * h2) & ss->bit_mask;
+ vbf[bit_loc>>b] |= 1U << (((bit_loc & (a - 1)) << shift) + set - 1);
+}
+
+
+
+int
+rte_member_lookup_vbf(const struct rte_member_setsum *ss, const void *key,
+ MEMBER_SET_TYPE *set_id)
+{
+ uint32_t j;
+ uint32_t h1 = MEMBER_PRIM_HASH(key, ss->key_len, ss->prim_hash_seed);
+ uint32_t h2 = MEMBER_SEC_HASH(&h1, 4, ss->sec_hash_seed);
+ uint32_t mask = 0xFFFFFFFF;
+ uint32_t shift = __builtin_ctz(ss->num_set);
+ uint32_t a = 32 >> shift;
+ uint32_t b = __builtin_ctz(a);
+
+ for (j = 0; j < ss->num_hashes; j++)
+ mask &= test_bit(h1, h2, j, shift, a, b, ss);
+
+ if (mask) {
+ *set_id = __builtin_ctz(mask) + 1;
+ return 1;
+ }
+
+ *set_id = RTE_MEMBER_NO_MATCH;
+ return 0;
+}
+
+
+
+uint32_t
+rte_member_lookup_bulk_vbf(const struct rte_member_setsum *ss,
+ const void **keys, uint32_t num_keys, MEMBER_SET_TYPE *set_ids)
+{
+ uint32_t i, k, h1, h2;
+ uint32_t ret = 0;
+ uint32_t mask;
+ uint32_t shift = __builtin_ctz(ss->num_set);
+ uint32_t a = 32 >> shift;
+ uint32_t b = __builtin_ctz(a);
+
+ for (i = 0; i < num_keys; i++) {
+ mask = 0xFFFFFFFF;
+ h1 = MEMBER_PRIM_HASH(keys[i], ss->key_len,
+ ss->prim_hash_seed);
+ h2 = MEMBER_SEC_HASH(&h1, 4, ss->sec_hash_seed);
+ for (k = 0; k < ss->num_hashes; k++)
+ mask &= test_bit(h1, h2, k, shift, a, b, ss);
+
+ if (mask) {
+ set_ids[i] = __builtin_ctz(mask) + 1;
+ ret++;
+ } else
+ set_ids[i] = RTE_MEMBER_NO_MATCH;
+ }
+ return ret;
+}
+
+
+
+uint32_t
+rte_member_lookup_multi_vbf(const struct rte_member_setsum *ss,
+ const void *key, uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id)
+{
+ uint32_t ret = 0;
+ uint32_t j;
+ uint32_t h1 = MEMBER_PRIM_HASH(key, ss->key_len, ss->prim_hash_seed);
+ uint32_t h2 = MEMBER_SEC_HASH(&h1, 4, ss->sec_hash_seed);
+ uint32_t mask = 0xFFFFFFFF;
+ uint32_t shift = __builtin_ctz(ss->num_set);
+ uint32_t a = 32 >> shift;
+ uint32_t b = __builtin_ctz(a);
+
+ for (j = 0; j < ss->num_hashes; j++)
+ mask &= test_bit(h1, h2, j, shift, a, b, ss);
+
+ while (mask) {
+ uint32_t loc = __builtin_ctz(mask);
+ set_id[ret] = loc + 1;
+ ret++;
+ if (ret >= match_per_key)
+ return ret;
+ mask &= ~(1U << loc);
+ }
+ return ret;
+}
+
+
+
+uint32_t
+rte_member_lookup_multi_bulk_vbf(const struct rte_member_setsum *ss,
+ const void **keys, uint32_t num_keys, uint32_t match_per_key,
+ uint32_t *match_count,
+ MEMBER_SET_TYPE *set_ids)
+{
+ uint32_t i, k, h1, h2;
+ uint32_t ret = 0;
+ uint32_t match_cnt_t;
+ uint32_t mask;
+ uint32_t shift = __builtin_ctz(ss->num_set);
+ uint32_t a = 32 >> shift;
+ uint32_t b = __builtin_ctz(a);
+
+ for (i = 0; i < num_keys; i++) {
+ match_cnt_t = 0;
+ mask = 0xFFFFFFFF;
+ h1 = MEMBER_PRIM_HASH(keys[i], ss->key_len,
+ ss->prim_hash_seed);
+ h2 = MEMBER_SEC_HASH(&h1, 4, ss->sec_hash_seed);
+
+ for (k = 0; k < ss->num_hashes; k++)
+ mask &= test_bit(h1, h2, k, shift, a, b, ss);
+
+ while (mask) {
+ uint32_t loc = __builtin_ctz(mask);
+ if (match_cnt_t >= match_per_key)
+ break;
+ set_ids[i * match_per_key + match_cnt_t] = loc + 1;
+ match_cnt_t++;
+ mask &= ~(1U << loc);
+ }
+ match_count[i] = match_cnt_t;
+ if (match_cnt_t != 0)
+ ret++;
+ }
+ return ret;
+}
+
+
+int
+rte_member_add_vbf(const struct rte_member_setsum *ss,
+ const void *key, MEMBER_SET_TYPE set_id)
+{
+ uint32_t i, h1, h2;
+
+ if (set_id > ss->num_set || set_id == RTE_MEMBER_NO_MATCH)
+ return -EINVAL;
+
+ h1 = MEMBER_PRIM_HASH(key, ss->key_len, ss->prim_hash_seed);
+ h2 = MEMBER_SEC_HASH(&h1, 4, ss->sec_hash_seed);
+ uint32_t shift = __builtin_ctz(ss->num_set);
+ uint32_t a = 32 >> shift;
+ uint32_t b = __builtin_ctz(a);
+
+ for (i = 0; i < ss->num_hashes; i++)
+ set_bit(h1, h2, i, shift, a, b, ss, set_id);
+ return 0;
+}
+
+void
+rte_member_free_vbf(struct rte_member_setsum *ss)
+{
+ rte_free(ss->table);
+}
+
+
+void
+rte_member_reset_vbf(const struct rte_member_setsum *ss)
+{
+ uint32_t *vbf = ss->table;
+ memset(vbf, 0, (ss->num_set * ss->bits) >> 3);
+}
diff --git a/lib/librte_member/rte_member_vbf.h b/lib/librte_member/rte_member_vbf.h
new file mode 100644
index 0000000..3d3ee33
--- /dev/null
+++ b/lib/librte_member/rte_member_vbf.h
@@ -0,0 +1,85 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _RTE_MEMBER_VBF_H_
+#define _RTE_MEMBER_VBF_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+int
+rte_member_create_vbf(struct rte_member_setsum *ss,
+ const struct rte_member_parameters *params);
+
+
+int
+rte_member_lookup_vbf(const struct rte_member_setsum *setsum,
+ const void *key, MEMBER_SET_TYPE *set_id);
+
+
+uint32_t
+rte_member_lookup_bulk_vbf(const struct rte_member_setsum *setsum,
+ const void **keys, uint32_t num_keys,
+ MEMBER_SET_TYPE *set_ids);
+
+
+uint32_t
+rte_member_lookup_multi_vbf(const struct rte_member_setsum *setsum,
+ const void *key, uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id);
+
+
+uint32_t
+rte_member_lookup_multi_bulk_vbf(const struct rte_member_setsum *setsum,
+ const void **keys, uint32_t num_keys, uint32_t match_per_key,
+ uint32_t *match_count,
+ MEMBER_SET_TYPE *set_ids);
+
+int
+rte_member_add_vbf(const struct rte_member_setsum *setsum,
+ const void *key, MEMBER_SET_TYPE set_id);
+
+void
+rte_member_free_vbf(struct rte_member_setsum *ss);
+
+
+void
+rte_member_reset_vbf(const struct rte_member_setsum *setsum);
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_MEMBER_VBF_H_ */
--
2.7.4
Yipeng Wang
2017-08-22 00:19:50 UTC
Permalink
For key search, the signatures of all entries are compared against
the signature of the key that is being looked up. Since all
signatures are contguously put in a bucket, they can be compared
with vector instructions (AVX2), achieving higher lookup performance.

This patch adds AVX2 implementation in a separate header file.

Signed-off-by: Yipeng Wang <***@intel.com>
---
lib/librte_member/rte_member_ht.c | 143 ++++++++++++++++++++++++++++---------
lib/librte_member/rte_member_x86.h | 111 ++++++++++++++++++++++++++++
2 files changed, 222 insertions(+), 32 deletions(-)
create mode 100644 lib/librte_member/rte_member_x86.h

diff --git a/lib/librte_member/rte_member_ht.c b/lib/librte_member/rte_member_ht.c
index 3e411ed..2f52220 100644
--- a/lib/librte_member/rte_member_ht.c
+++ b/lib/librte_member/rte_member_ht.c
@@ -40,6 +40,10 @@
#include "rte_member.h"
#include "rte_member_ht.h"

+#if defined(RTE_ARCH_X86)
+#include "rte_member_x86.h"
+#endif
+

static inline int
insert_overwrite_search(uint32_t bucket, SIG_TYPE tmp_sig,
@@ -133,6 +137,13 @@ rte_member_create_ht(struct rte_member_setsum *ss,
for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++)
buckets[i].sets[j] = RTE_MEMBER_NO_MATCH;
}
+#if defined(RTE_ARCH_X86)
+ if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2) &&
+ RTE_MEMBER_BUCKET_ENTRIES == 16)
+ ss->sig_cmp_fn = RTE_MEMBER_COMPARE_AVX2;
+ else
+#endif
+ ss->sig_cmp_fn = RTE_MEMBER_COMPARE_SCALAR;


RTE_LOG(DEBUG, MEMBER, "Hash table based filter created, "
@@ -172,11 +183,23 @@ rte_member_lookup_ht(const struct rte_member_setsum *ss,
*set_id = RTE_MEMBER_NO_MATCH;
get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);

- if (search_bucket_single(prim_bucket, tmp_sig, buckets,
- set_id) ||
- search_bucket_single(sec_bucket, tmp_sig,
- buckets, set_id))
- return 1;
+ switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+ case RTE_MEMBER_COMPARE_AVX2:
+ if (search_bucket_single_avx(prim_bucket, tmp_sig, buckets,
+ set_id) ||
+ search_bucket_single_avx(sec_bucket, tmp_sig,
+ buckets, set_id))
+ return 1;
+ break;
+#endif
+ default:
+ if (search_bucket_single(prim_bucket, tmp_sig, buckets,
+ set_id) ||
+ search_bucket_single(sec_bucket, tmp_sig,
+ buckets, set_id))
+ return 1;
+ }

return 0;
}
@@ -201,13 +224,27 @@ rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss,
}

for (i = 0; i < num_keys; i++) {
- if (search_bucket_single(prim_buckets[i], tmp_sig[i],
- buckets, &set_id[i]) ||
- search_bucket_single(sec_buckets[i],
- tmp_sig[i], buckets, &set_id[i]))
- ret++;
- else
- set_id[i] = RTE_MEMBER_NO_MATCH;
+ switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+ case RTE_MEMBER_COMPARE_AVX2:
+ if (search_bucket_single_avx(prim_buckets[i],
+ tmp_sig[i], buckets, &set_id[i]) ||
+ search_bucket_single_avx(sec_buckets[i],
+ tmp_sig[i], buckets, &set_id[i]))
+ ret++;
+ else
+ set_id[i] = RTE_MEMBER_NO_MATCH;
+ break;
+#endif
+ default:
+ if (search_bucket_single(prim_buckets[i], tmp_sig[i],
+ buckets, &set_id[i]) ||
+ search_bucket_single(sec_buckets[i],
+ tmp_sig[i], buckets, &set_id[i]))
+ ret++;
+ else
+ set_id[i] = RTE_MEMBER_NO_MATCH;
+ }
}
return ret;
}
@@ -225,12 +262,24 @@ rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,

get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);

- search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
- match_per_key, set_id);
- if (ret < match_per_key)
- search_bucket_multi(sec_bucket, tmp_sig,
- buckets, &ret, match_per_key, set_id);
- return ret;
+ switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+ case RTE_MEMBER_COMPARE_AVX2:
+ search_bucket_multi_avx(prim_bucket, tmp_sig, buckets,
+ &ret, match_per_key, set_id);
+ if (ret < match_per_key)
+ search_bucket_multi_avx(sec_bucket, tmp_sig,
+ buckets, &ret, match_per_key, set_id);
+ return ret;
+#endif
+ default:
+ search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
+ match_per_key, set_id);
+ if (ret < match_per_key)
+ search_bucket_multi(sec_bucket, tmp_sig,
+ buckets, &ret, match_per_key, set_id);
+ return ret;
+ }
}


@@ -257,16 +306,34 @@ rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,
for (i = 0; i < num_keys; i++) {
match_cnt_t = 0;

- search_bucket_multi(prim_buckets[i], tmp_sig[i],
- buckets, &match_cnt_t, match_per_key,
- &set_ids[i*match_per_key]);
- if (match_cnt_t < match_per_key)
- search_bucket_multi(sec_buckets[i], tmp_sig[i],
+ switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+ case RTE_MEMBER_COMPARE_AVX2:
+ search_bucket_multi_avx(prim_buckets[i], tmp_sig[i],
buckets, &match_cnt_t, match_per_key,
&set_ids[i*match_per_key]);
- match_count[i] = match_cnt_t;
- if (match_cnt_t != 0)
- ret++;
+ if (match_cnt_t < match_per_key)
+ search_bucket_multi_avx(sec_buckets[i],
+ tmp_sig[i], buckets, &match_cnt_t,
+ match_per_key,
+ &set_ids[i*match_per_key]);
+ match_count[i] = match_cnt_t;
+ if (match_cnt_t != 0)
+ ret++;
+ break;
+#endif
+ default:
+ search_bucket_multi(prim_buckets[i], tmp_sig[i],
+ buckets, &match_cnt_t, match_per_key,
+ &set_ids[i*match_per_key]);
+ if (match_cnt_t < match_per_key)
+ search_bucket_multi(sec_buckets[i], tmp_sig[i],
+ buckets, &match_cnt_t, match_per_key,
+ &set_ids[i*match_per_key]);
+ match_count[i] = match_cnt_t;
+ if (match_cnt_t != 0)
+ ret++;
+ }
}
return ret;
}
@@ -298,12 +365,24 @@ try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,

static inline int
try_overwrite(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
- SIG_TYPE sig, MEMBER_SET_TYPE set_id)
+ SIG_TYPE sig, MEMBER_SET_TYPE set_id,
+ enum rte_member_sig_compare_function cmp_fn)
{
- if (insert_overwrite_search(prim, sig, buckets, set_id) ||
- insert_overwrite_search(sec, sig, buckets,
- set_id))
- return 0;
+ switch (cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+ case RTE_MEMBER_COMPARE_AVX2:
+ if (insert_overwrite_search_avx(prim, sig, buckets, set_id) ||
+ insert_overwrite_search_avx(sec, sig, buckets,
+ set_id))
+ return 0;
+ break;
+#endif
+ default:
+ if (insert_overwrite_search(prim, sig, buckets, set_id) ||
+ insert_overwrite_search(sec, sig, buckets,
+ set_id))
+ return 0;
+ }
return -1;
}

@@ -409,7 +488,7 @@ rte_member_add_ht(const struct rte_member_setsum *ss,
/* if it is cache based filter, we try overwriting existing entry */
if (ss->cache) {
ret = try_overwrite(buckets, prim_bucket, sec_bucket, tmp_sig,
- set_id);
+ set_id, ss->sig_cmp_fn);
if (ret != -1)
return ret;
}
diff --git a/lib/librte_member/rte_member_x86.h b/lib/librte_member/rte_member_x86.h
new file mode 100644
index 0000000..c55f128
--- /dev/null
+++ b/lib/librte_member/rte_member_x86.h
@@ -0,0 +1,111 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _RTE_MEMBER_X86_H_
+#define _RTE_MEMBER_X86_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <x86intrin.h>
+
+
+#if defined(RTE_MACHINE_CPUFLAG_AVX2)
+
+
+static inline int
+insert_overwrite_search_avx(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ MEMBER_SET_TYPE set_id)
+{
+ uint32_t hitmask = _mm256_movemask_epi8((__m256i)_mm256_cmpeq_epi16(
+ _mm256_load_si256((__m256i const *)buckets[bucket].sigs),
+ _mm256_set1_epi16(tmp_sig)));
+ if (hitmask) {
+ uint32_t hit_idx = __builtin_ctzl(hitmask) / 2;
+ buckets[bucket].sets[hit_idx] = set_id;
+ return 1;
+ }
+ return 0;
+}
+
+
+static inline int
+search_bucket_single_avx(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ MEMBER_SET_TYPE *set_id)
+{
+ uint32_t hitmask = _mm256_movemask_epi8((__m256i)_mm256_cmpeq_epi16(
+ _mm256_load_si256((__m256i const *)buckets[bucket].sigs),
+ _mm256_set1_epi16(tmp_sig)));
+ while (hitmask) {
+ uint32_t hit_idx = __builtin_ctzl(hitmask) / 2;
+ if (buckets[bucket].sets[hit_idx] != RTE_MEMBER_NO_MATCH) {
+ *set_id = buckets[bucket].sets[hit_idx];
+ return 1;
+ }
+ hitmask &= ~(3U << (hit_idx) * 2);
+ }
+ return 0;
+}
+
+static inline void
+search_bucket_multi_avx(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ uint32_t *counter,
+ uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id)
+{
+ uint32_t hitmask = _mm256_movemask_epi8((__m256i)_mm256_cmpeq_epi16(
+ _mm256_load_si256((__m256i const *)buckets[bucket].sigs),
+ _mm256_set1_epi16(tmp_sig)));
+ while (hitmask) {
+ uint32_t hit_idx = __builtin_ctzl(hitmask) / 2;
+ if (buckets[bucket].sets[hit_idx] != RTE_MEMBER_NO_MATCH) {
+ set_id[*counter] = buckets[bucket].sets[hit_idx];
+ (*counter)++;
+ if (*counter >= match_per_key)
+ return;
+ }
+ hitmask &= ~(3U << (hit_idx) * 2);
+ }
+}
+#endif
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_MEMBER_X86_H_ */
--
2.7.4
Yipeng Wang
2017-08-22 00:19:51 UTC
Permalink
This patch enables the Membership library.

Signed-off-by: Yipeng Wang <***@intel.com>
---
MAINTAINERS | 7 +++++++
config/common_base | 5 +++++
lib/librte_member/Makefile | 2 ++
mk/rte.app.mk | 2 ++
4 files changed, 16 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index a0cd75e..e372edf 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -734,6 +734,13 @@ F: test/test/test_meter.c
F: examples/qos_meter/
F: doc/guides/sample_app_ug/qos_metering.rst

+Membership - EXPERIMENTAL
+M: Yipeng Wang <***@intel.com>
+M: Sameh Gobriel <***@intel.com>
+F: lib/librte_member/
+F: doc/guides/prog_guide/member_lib.rst
+F: test/test/test_member*
+

Other libraries
---------------
diff --git a/config/common_base b/config/common_base
index 5e97a08..5e31ced 100644
--- a/config/common_base
+++ b/config/common_base
@@ -595,6 +595,11 @@ CONFIG_RTE_LIBRTE_HASH_DEBUG=n
CONFIG_RTE_LIBRTE_EFD=y

#
+# Compile librte_member
+#
+CONFIG_RTE_LIBRTE_MEMBER=y
+
+#
# Compile librte_jobstats
#
CONFIG_RTE_LIBRTE_JOBSTATS=y
diff --git a/lib/librte_member/Makefile b/lib/librte_member/Makefile
index 90bdfe8..3c2c4e0 100644
--- a/lib/librte_member/Makefile
+++ b/lib/librte_member/Makefile
@@ -36,6 +36,8 @@ LIB = librte_member.a

CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3

+LDLIBS += -lm
+
EXPORT_MAP := rte_member_version.map

LIBABIVER := 1
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index c25fdd9..c79acf0 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -86,6 +86,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CFGFILE) += -lrte_cfgfile
_LDLIBS-y += --whole-archive

_LDLIBS-$(CONFIG_RTE_LIBRTE_HASH) += -lrte_hash
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lrte_member
_LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lrte_vhost
_LDLIBS-$(CONFIG_RTE_LIBRTE_KVARGS) += -lrte_kvargs
_LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF) += -lrte_mbuf
@@ -196,6 +197,7 @@ endif
_LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lm
_LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrt
_LDLIBS-$(CONFIG_RTE_LIBRTE_METER) += -lm
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lm
ifeq ($(CONFIG_RTE_LIBRTE_VHOST_NUMA),y)
_LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lnuma
endif
--
2.7.4
Yipeng Wang
2017-08-22 00:19:52 UTC
Permalink
This patch adds functional and performance tests for membership
library.

Signed-off-by: Yipeng Wang <***@intel.com>
---
test/test/Makefile | 3 +
test/test/test_member.c | 550 ++++++++++++++++++++++++++++++++++++
test/test/test_member_perf.c | 643 +++++++++++++++++++++++++++++++++++++++++++
3 files changed, 1196 insertions(+)
create mode 100644 test/test/test_member.c
create mode 100644 test/test/test_member_perf.c

diff --git a/test/test/Makefile b/test/test/Makefile
index 42d9a49..b61dde3 100644
--- a/test/test/Makefile
+++ b/test/test/Makefile
@@ -123,6 +123,9 @@ SRCS-y += test_logs.c
SRCS-y += test_memcpy.c
SRCS-y += test_memcpy_perf.c

+
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member_perf.c
SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd.c
SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd_perf.c

diff --git a/test/test/test_member.c b/test/test/test_member.c
new file mode 100644
index 0000000..470e80a
--- /dev/null
+++ b/test/test/test_member.c
@@ -0,0 +1,550 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+/* This test is for member library's simple feature test */
+
+#include <rte_memcpy.h>
+#include <rte_malloc.h>
+#include <rte_member.h>
+#include <rte_byteorder.h>
+#include <rte_random.h>
+#include <rte_debug.h>
+#include <rte_ip.h>
+
+#include "test.h"
+
+void *setsum_ht;
+void *setsum_cache;
+void *setsum_vbf;
+
+/* 5-tuple key type */
+struct flow_key {
+ uint32_t ip_src;
+ uint32_t ip_dst;
+ uint16_t port_src;
+ uint16_t port_dst;
+ uint8_t proto;
+} __attribute__((packed));
+
+
+/* Keys used by unit test functions */
+static struct flow_key keys[5] = {
+ {
+ .ip_src = IPv4(0x03, 0x02, 0x01, 0x00),
+ .ip_dst = IPv4(0x07, 0x06, 0x05, 0x04),
+ .port_src = 0x0908,
+ .port_dst = 0x0b0a,
+ .proto = 0x0c,
+ },
+ {
+ .ip_src = IPv4(0x13, 0x12, 0x11, 0x10),
+ .ip_dst = IPv4(0x17, 0x16, 0x15, 0x14),
+ .port_src = 0x1918,
+ .port_dst = 0x1b1a,
+ .proto = 0x1c,
+ },
+ {
+ .ip_src = IPv4(0x23, 0x22, 0x21, 0x20),
+ .ip_dst = IPv4(0x27, 0x26, 0x25, 0x24),
+ .port_src = 0x2928,
+ .port_dst = 0x2b2a,
+ .proto = 0x2c,
+ },
+ {
+ .ip_src = IPv4(0x33, 0x32, 0x31, 0x30),
+ .ip_dst = IPv4(0x37, 0x36, 0x35, 0x34),
+ .port_src = 0x3938,
+ .port_dst = 0x3b3a,
+ .proto = 0x3c,
+ },
+ {
+ .ip_src = IPv4(0x43, 0x42, 0x41, 0x40),
+ .ip_dst = IPv4(0x47, 0x46, 0x45, 0x44),
+ .port_src = 0x4948,
+ .port_dst = 0x4b4a,
+ .proto = 0x4c,
+ }
+};
+
+uint32_t test_set[5] = {1, 2, 3, 4, 5};
+
+#define ITERATIONS 3
+#define KEY_SIZE 4
+
+#define MAX_ENTRIES (1 << 16)
+uint8_t gened_keys[MAX_ENTRIES][KEY_SIZE];
+
+static struct rte_member_parameters params = {
+ .num_keys = MAX_ENTRIES, /* Total hash table entries. */
+ .key_len = KEY_SIZE, /* Length of hash key. */
+
+ /*num_set and false_positive_rate only relevant to vBF setsum*/
+ .num_set = 32,
+ .false_positive_rate = 0.03,
+ .prim_hash_seed = 1,
+ .sec_hash_seed = 11,
+ .socket_id = 1 /* NUMA Socket ID for memory. */
+};
+
+
+/* Create test setsummaries. */
+static int test_member_create(void)
+{
+ params.key_len = sizeof(struct flow_key);
+
+ params.name = "test_member_ht";
+ params.iscache = 0;
+ params.type = RTE_MEMBER_TYPE_HT;
+ setsum_ht = rte_member_create(&params);
+
+ params.name = "test_member_cache";
+ params.iscache = 1;
+ setsum_cache = rte_member_create(&params);
+
+ params.name = "test_member_vbf";
+ params.type = RTE_MEMBER_TYPE_VBF;
+ setsum_vbf = rte_member_create(&params);
+
+ if (setsum_ht == NULL || setsum_cache == NULL || setsum_vbf == NULL) {
+ printf("Creation of setsums fail\n");
+ return -1;
+ }
+ printf("Creation of setsums success\n");
+ return 0;
+}
+
+static int test_member_insert(void)
+{
+ int ret_ht, ret_cache, ret_vbf, i;
+
+ for (i = 0; i < 5; i++) {
+ ret_ht = rte_member_add(setsum_ht, &keys[i], test_set[i]);
+ ret_cache = rte_member_add(setsum_cache, &keys[i],
+ test_set[i]);
+ ret_vbf = rte_member_add(setsum_vbf, &keys[i], test_set[i]);
+ TEST_ASSERT(ret_ht >= 0 && ret_cache >= 0 && ret_vbf >= 0,
+ "insert error");
+ }
+ printf("insert key success\n");
+ return 0;
+}
+
+static int test_member_lookup(void)
+{
+ int ret_ht, ret_cache, ret_vbf, i;
+ uint16_t set_ht, set_cache, set_vbf;
+ MEMBER_SET_TYPE set_ids_ht[5] = {0};
+ MEMBER_SET_TYPE set_ids_cache[5] = {0};
+ MEMBER_SET_TYPE set_ids_vbf[5] = {0};
+
+ uint32_t num_key_ht = 5;
+ uint32_t num_key_cache = 5;
+ uint32_t num_key_vbf = 5;
+
+ const void *key_array[5];
+
+ /* single lookup test */
+ for (i = 0; i < 5; i++) {
+ ret_ht = rte_member_lookup(setsum_ht, &keys[i], &set_ht);
+ ret_cache = rte_member_lookup(setsum_cache, &keys[i],
+ &set_cache);
+ ret_vbf = rte_member_lookup(setsum_vbf, &keys[i], &set_vbf);
+ TEST_ASSERT(ret_ht >= 0 && ret_cache >= 0 && ret_vbf >= 0,
+ "single lookup function error");
+
+ TEST_ASSERT(set_ht == test_set[i] &&
+ set_cache == test_set[i] &&
+ set_vbf == test_set[i],
+ "single lookup set value error");
+ }
+ printf("lookup single key success\n");
+
+ /* bulk lookup test */
+ for (i = 0; i < 5; i++)
+ key_array[i] = &keys[i];
+
+ ret_ht = rte_member_lookup_bulk(setsum_ht, &key_array[0],
+ num_key_ht, set_ids_ht);
+
+ ret_cache = rte_member_lookup_bulk(setsum_cache, &key_array[0],
+ num_key_cache, set_ids_cache);
+
+ ret_vbf = rte_member_lookup_bulk(setsum_vbf, &key_array[0],
+ num_key_vbf, set_ids_vbf);
+
+ TEST_ASSERT(ret_ht >= 0 && ret_cache >= 0 && ret_vbf >= 0,
+ "bulk lookup function error");
+
+ for (i = 0; i < 5; i++) {
+ TEST_ASSERT((set_ids_ht[i] == test_set[i]) &&
+ (set_ids_cache[i] == test_set[i]) &&
+ (set_ids_vbf[i] == test_set[i]),
+ "bulk lookup result error");
+ }
+
+ return 0;
+}
+
+
+static int test_member_delete(void)
+{
+ int ret_ht, ret_cache, ret_vbf, i;
+ uint16_t set_ht, set_cache, set_vbf;
+ for (i = 0; i < 5; i++) {
+ ret_ht = rte_member_delete(setsum_ht, &keys[i], test_set[i]);
+ ret_cache = rte_member_delete(setsum_cache, &keys[i],
+ test_set[i]);
+ ret_vbf = rte_member_delete(setsum_vbf, &keys[i], test_set[i]);
+ /* VBF does not support delete yet, so return error code */
+ TEST_ASSERT(ret_ht >= 0 && ret_cache >= 0,
+ "key deletion function error");
+ TEST_ASSERT(ret_vbf < 0,
+ "vbf does not support deletion, error");
+ }
+
+ for (i = 0; i < 5; i++) {
+ ret_ht = rte_member_lookup(setsum_ht, &keys[i], &set_ht);
+ ret_cache = rte_member_lookup(setsum_cache, &keys[i],
+ &set_cache);
+ ret_vbf = rte_member_lookup(setsum_vbf, &keys[i], &set_vbf);
+ TEST_ASSERT(ret_ht >= 0 && ret_cache >= 0,
+ "key lookup function error");
+ TEST_ASSERT(set_ht == RTE_MEMBER_NO_MATCH &&
+ ret_cache == RTE_MEMBER_NO_MATCH,
+ "key deletion failed");
+ }
+ printf("delete success\n");
+ return 0;
+}
+
+
+static int test_member_multimatch(void)
+{
+ int ret_ht, ret_vbf, ret_cache;
+ MEMBER_SET_TYPE set_ids_ht[32] = {0};
+ MEMBER_SET_TYPE set_ids_vbf[32] = {0};
+ MEMBER_SET_TYPE set_ids_cache[32] = {0};
+
+ MEMBER_SET_TYPE set_ids_ht_m[5][32] = {{0} };
+ MEMBER_SET_TYPE set_ids_vbf_m[5][32] = {{0} };
+ MEMBER_SET_TYPE set_ids_cache_m[5][32] = {{0} };
+
+ uint32_t match_count_ht[5];
+ uint32_t match_count_vbf[5];
+ uint32_t match_count_cache[5];
+
+ uint32_t num_key_ht = 5;
+ uint32_t num_key_vbf = 5;
+ uint32_t num_key_cache = 5;
+
+ const void *key_array[5];
+
+ uint32_t i, j;
+ /* same key at most inserted 2*entry_per_bucket times for HT mode */
+ for (i = 1; i < 33; i++) {
+ for (j = 0; j < 5; j++) {
+ ret_ht = rte_member_add(setsum_ht, &keys[j], i);
+ ret_vbf = rte_member_add(setsum_vbf, &keys[j], i);
+ ret_cache = rte_member_add(setsum_cache, &keys[j], i);
+
+ TEST_ASSERT(ret_ht >= 0 && ret_vbf >= 0 &&
+ ret_cache >= 0,
+ "insert function error");
+ }
+ }
+
+ /* single multimatch test */
+ for (i = 0; i < 5; i++) {
+ ret_vbf = rte_member_lookup_multi(setsum_vbf, &keys[i], 32,
+ set_ids_vbf);
+ ret_ht = rte_member_lookup_multi(setsum_ht, &keys[i], 32,
+ set_ids_ht);
+ ret_cache = rte_member_lookup_multi(setsum_cache, &keys[i], 32,
+ set_ids_cache);
+ /*
+ * for cache mode, it does not support multimatch
+ * the mutimatch should work like single match
+ */
+ TEST_ASSERT(ret_ht == 32 && ret_vbf == 32 && ret_cache == 1,
+ "single lookup_multi error");
+ TEST_ASSERT(set_ids_cache[0] == 32,
+ "single lookup_multi cache error");
+
+ for (j = 1; j < 33; j++) {
+ TEST_ASSERT(set_ids_ht[j-1] == j &&
+ set_ids_vbf[j-1] == j,
+ "single multimatch lookup error");
+ }
+ }
+ printf("lookup single key for multimatch success\n");
+
+ /* bulk multimatch test */
+
+ for (i = 0; i < 5; i++)
+ key_array[i] = &keys[i];
+ ret_vbf = rte_member_lookup_multi_bulk(setsum_vbf,
+ &key_array[0], num_key_ht, 32, match_count_vbf,
+ (MEMBER_SET_TYPE *)set_ids_vbf_m);
+
+ ret_ht = rte_member_lookup_multi_bulk(setsum_ht,
+ &key_array[0], num_key_vbf, 32, match_count_ht,
+ (MEMBER_SET_TYPE *)set_ids_ht_m);
+
+ ret_cache = rte_member_lookup_multi_bulk(setsum_cache,
+ &key_array[0], num_key_cache, 32, match_count_cache,
+ (MEMBER_SET_TYPE *)set_ids_cache_m);
+
+
+ for (j = 0; j < 5; j++) {
+ TEST_ASSERT(match_count_ht[j] == 32,
+ "bulk multimatch lookup HT match count error");
+ TEST_ASSERT(match_count_vbf[j] == 32,
+ "bulk multimatch lookup vBF match count error");
+ TEST_ASSERT(match_count_cache[j] == 1,
+ "bulk multimatch lookup CACHE match count error");
+ TEST_ASSERT(set_ids_cache_m[j][0] == 32,
+ "bulk multimatch lookup CACHE set value error");
+
+ for (i = 1; i < 33; i++) {
+ TEST_ASSERT(set_ids_ht_m[j][i-1] == i,
+ "bulk multimatch lookup HT set value error");
+ TEST_ASSERT(set_ids_vbf_m[j][i-1] == i,
+ "bulk multimatch lookup vBF set value error");
+ }
+ }
+
+ printf("lookup for bulk multimatch success\n");
+
+ return 0;
+}
+
+
+static int key_compare(const void *key1, const void *key2)
+{
+ return memcmp(key1, key2, KEY_SIZE);
+}
+
+static void
+setup_keys_and_data(void)
+{
+ unsigned int i, j;
+ int num_duplicates;
+
+ /* Reset all arrays */
+ for (i = 0; i < KEY_SIZE; i++)
+ gened_keys[0][i] = 0;
+
+ /* Generate a list of keys, some of which may be duplicates */
+ for (i = 0; i < MAX_ENTRIES; i++) {
+ for (j = 0; j < KEY_SIZE; j++)
+ gened_keys[i][j] = rte_rand() & 0xFF;
+ }
+
+ /* Remove duplicates from the keys array */
+ do {
+ num_duplicates = 0;
+ /* Sort the list of keys to make it easier to find duplicates */
+ qsort(gened_keys, MAX_ENTRIES, KEY_SIZE, key_compare);
+
+ /* Sift through the list of keys and look for duplicates */
+ int num_duplicates = 0;
+ for (i = 0; i < MAX_ENTRIES - 1; i++) {
+ if (memcmp(gened_keys[i], gened_keys[i + 1],
+ KEY_SIZE) == 0) {
+ /* This key already exists, try again */
+ num_duplicates++;
+ for (j = 0; j < KEY_SIZE; j++)
+ gened_keys[i][j] = rte_rand() & 0xFF;
+ }
+ }
+ } while (num_duplicates != 0);
+}
+
+
+static inline int
+add_gened_keys(void *setsum, unsigned int *added_keys)
+{
+ int ret = 0;
+
+ for (*added_keys = 0; ret >= 0 && *added_keys < MAX_ENTRIES;
+ (*added_keys)++) {
+ uint16_t set = (rte_rand() & 0xf) + 1;
+ ret = rte_member_add(setsum, &gened_keys[*added_keys], set);
+ }
+ return ret;
+}
+
+
+static inline int
+add_gened_keys_cache(void *setsum, unsigned int *added_keys)
+{
+ int ret = 0;
+
+ for (*added_keys = 0; ret == 0 && *added_keys < MAX_ENTRIES;
+ (*added_keys)++) {
+ uint16_t set = (rte_rand() & 0xf) + 1;
+ ret = rte_member_add(setsum, &gened_keys[*added_keys], set);
+ }
+ return ret;
+}
+
+static int
+test_member_loadfactor(void)
+{
+ unsigned int j;
+ unsigned int added_keys, average_keys_added = 0;
+ int ret;
+
+ setup_keys_and_data();
+
+ rte_member_free(setsum_ht);
+ rte_member_free(setsum_cache);
+ rte_member_free(setsum_vbf);
+
+ params.key_len = KEY_SIZE;
+ params.name = "test_member_ht";
+ params.iscache = 0;
+ params.type = RTE_MEMBER_TYPE_HT;
+ setsum_ht = rte_member_create(&params);
+
+ params.name = "test_member_cache";
+ params.iscache = 1;
+ setsum_cache = rte_member_create(&params);
+
+
+ if (setsum_ht == NULL || setsum_cache == NULL) {
+ printf("Creation of setsums fail\n");
+ return -1;
+ }
+ /* test HT mode */
+ for (j = 0; j < ITERATIONS; j++) {
+ /* Add random entries until key cannot be added */
+ ret = add_gened_keys(setsum_ht, &added_keys);
+ if (ret != -ENOSPC) {
+ printf("Unexpected error when adding keys\n");
+ return -1;
+ }
+ average_keys_added += added_keys;
+
+ /* Reset the table */
+ rte_member_reset(setsum_ht);
+
+ /* Print a dot to show progress on operations */
+ printf(".");
+ fflush(stdout);
+ }
+
+ average_keys_added /= ITERATIONS;
+
+ printf("\nKeys inserted when no space(non-cache) = %.2f%% (%u/%u)\n",
+ ((double) average_keys_added / params.num_keys * 100),
+ average_keys_added, params.num_keys);
+
+ /* test cache mode */
+ added_keys = average_keys_added = 0;
+ for (j = 0; j < ITERATIONS; j++) {
+ /* Add random entries until key cannot be added */
+ ret = add_gened_keys_cache(setsum_cache, &added_keys);
+ if (ret != 1) {
+ printf("Unexpected error when adding keys\n");
+ return -1;
+ }
+ average_keys_added += added_keys;
+
+ /* Reset the table */
+ rte_member_reset(setsum_cache);
+
+ /* Print a dot to show progress on operations */
+ printf(".");
+ fflush(stdout);
+ }
+
+ average_keys_added /= ITERATIONS;
+
+ printf("\nKeys inserted when eviction happens(cache)= %.2f%% (%u/%u)\n",
+ ((double) average_keys_added / params.num_keys * 100),
+ average_keys_added, params.num_keys);
+ return 0;
+}
+
+
+
+static int
+test_member(void)
+{
+
+ /* Simple tests for initial debug usage */
+ if (test_member_create() < 0) {
+ rte_member_free(setsum_ht);
+ rte_member_free(setsum_cache);
+ rte_member_free(setsum_vbf);
+ return -1;
+ }
+ if (test_member_insert() < 0) {
+ rte_member_free(setsum_ht);
+ rte_member_free(setsum_cache);
+ rte_member_free(setsum_vbf);
+ return -1;
+ }
+ if (test_member_lookup() < 0) {
+ rte_member_free(setsum_ht);
+ rte_member_free(setsum_cache);
+ rte_member_free(setsum_vbf);
+ return -1;
+ }
+ if (test_member_delete() < 0) {
+ rte_member_free(setsum_ht);
+ rte_member_free(setsum_cache);
+ rte_member_free(setsum_vbf);
+ return -1;
+ }
+ if (test_member_multimatch() < 0) {
+ rte_member_free(setsum_ht);
+ rte_member_free(setsum_cache);
+ rte_member_free(setsum_vbf);
+ return -1;
+ }
+
+
+ if (test_member_loadfactor() < 0) {
+ rte_member_free(setsum_ht);
+ rte_member_free(setsum_cache);
+ }
+
+ rte_member_free(setsum_ht);
+ rte_member_free(setsum_cache);
+ rte_member_free(setsum_vbf);
+ return 0;
+}
+
+REGISTER_TEST_COMMAND(member_autotest, test_member);
diff --git a/test/test/test_member_perf.c b/test/test/test_member_perf.c
new file mode 100644
index 0000000..ba2b750
--- /dev/null
+++ b/test/test/test_member_perf.c
@@ -0,0 +1,643 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2016-2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <stdio.h>
+#include <inttypes.h>
+
+#include <rte_lcore.h>
+#include <rte_cycles.h>
+#include <rte_malloc.h>
+#include <rte_random.h>
+#include <rte_memcpy.h>
+#include <rte_thash.h>
+#include <rte_member.h>
+
+#include "test.h"
+
+#define NUM_KEYSIZES 10
+#define NUM_SHUFFLES 10
+#define MAX_KEYSIZE 64
+#define MAX_ENTRIES (1 << 19)
+#define KEYS_TO_ADD (MAX_ENTRIES * 75 / 100) /* 75% table utilization */
+#define NUM_LOOKUPS (KEYS_TO_ADD * 5) /* Loop among keys added, several times */
+#define VBF_SET_CNT 32
+#define BURST_SIZE 64
+#define VBF_FALSE_RATE 0.03
+
+
+static unsigned int test_socket_id;
+
+enum sstype {
+ HT = 0,
+ CACHE,
+ VBF,
+ NUM_TYPE
+};
+
+enum operations {
+ ADD = 0,
+ LOOKUP,
+ LOOKUP_BULK,
+ LOOKUP_MULTI,
+ LOOKUP_MULTI_BULK,
+ DELETE,
+ LOOKUP_MISS,
+ NUM_OPERATIONS
+};
+
+
+struct member_perf_params {
+ void *setsum[NUM_TYPE];
+ uint32_t key_size;
+ unsigned int cycle;
+};
+
+
+static uint32_t hashtest_key_lens[] = {
+ /* standard key sizes */
+ 4, 8, 16, 32, 48, 64,
+ /* IPv4 SRC + DST + protocol, unpadded */
+ 9,
+ /* IPv4 5-tuple, unpadded */
+ 13,
+ /* IPv6 5-tuple, unpadded */
+ 37,
+ /* IPv6 5-tuple, padded to 8-byte boundary */
+ 40
+};
+
+/* Array to store number of cycles per operation */
+uint64_t cycles[NUM_TYPE][NUM_KEYSIZES][NUM_OPERATIONS];
+uint64_t false_data[NUM_TYPE][NUM_KEYSIZES];
+uint64_t false_data_bulk[NUM_TYPE][NUM_KEYSIZES];
+uint64_t false_data_multi[NUM_TYPE][NUM_KEYSIZES];
+uint64_t false_data_multi_bulk[NUM_TYPE][NUM_KEYSIZES];
+
+uint64_t false_hit[NUM_TYPE][NUM_KEYSIZES];
+
+
+MEMBER_SET_TYPE data[NUM_TYPE][/* Array to store the data */KEYS_TO_ADD];
+
+/* Array to store all input keys */
+uint8_t keys[KEYS_TO_ADD][MAX_KEYSIZE];
+
+/* Shuffle the keys that have been added, so lookups will be totally random */
+static void
+shuffle_input_keys(struct member_perf_params *params)
+{
+ MEMBER_SET_TYPE temp_data;
+ unsigned int i, j;
+ uint32_t swap_idx;
+ uint8_t temp_key[MAX_KEYSIZE];
+
+ for (i = KEYS_TO_ADD - 1; i > 0; i--) {
+ swap_idx = rte_rand() % i;
+ memcpy(temp_key, keys[i], hashtest_key_lens[params->cycle]);
+ memcpy(keys[i], keys[swap_idx],
+ hashtest_key_lens[params->cycle]);
+ memcpy(keys[swap_idx], temp_key,
+ hashtest_key_lens[params->cycle]);
+ for (j = 0; j < NUM_TYPE; j++) {
+ temp_data = data[j][i];
+ data[j][i] = data[j][swap_idx];
+ data[j][swap_idx] = temp_data;
+ }
+ }
+}
+
+static int key_compare(const void *key1, const void *key2)
+{
+ return memcmp(key1, key2, MAX_KEYSIZE);
+}
+
+struct rte_member_parameters member_params = {
+ .num_keys = MAX_ENTRIES, /* Total hash table entries. */
+ .key_len = 4, /* Length of hash key. */
+
+ /* num_set and false_positive_rate only relevant to vBF setsum*/
+ .num_set = VBF_SET_CNT,
+ .false_positive_rate = 0.03,
+ .prim_hash_seed = 0,
+ .sec_hash_seed = 1,
+ .socket_id = 1, /* NUMA Socket ID for memory. */
+ };
+
+
+
+static int
+setup_keys_and_data(struct member_perf_params *params, unsigned int cycle,
+ int miss)
+{
+ unsigned int i, j;
+ int num_duplicates;
+
+ params->key_size = hashtest_key_lens[cycle];
+ params->cycle = cycle;
+
+ /* Reset all arrays */
+ for (i = 0; i < params->key_size; i++)
+ keys[0][i] = 0;
+
+ /* Generate a list of keys, some of which may be duplicates */
+ for (i = 0; i < KEYS_TO_ADD; i++) {
+ for (j = 0; j < params->key_size; j++)
+ keys[i][j] = rte_rand() & 0xFF;
+
+ data[HT][i] = data[CACHE][i] = (rte_rand() & 0x7FFE) + 1;
+ data[VBF][i] = rte_rand() % VBF_SET_CNT + 1;
+ }
+
+ /* Remove duplicates from the keys array */
+ do {
+ num_duplicates = 0;
+
+ /* Sort the list of keys to make it easier to find duplicates */
+ qsort(keys, KEYS_TO_ADD, MAX_KEYSIZE, key_compare);
+
+ /* Sift through the list of keys and look for duplicates */
+ int num_duplicates = 0;
+ for (i = 0; i < KEYS_TO_ADD - 1; i++) {
+ if (memcmp(keys[i], keys[i + 1],
+ params->key_size) == 0) {
+ /* This key already exists, try again */
+ num_duplicates++;
+ for (j = 0; j < params->key_size; j++)
+ keys[i][j] = rte_rand() & 0xFF;
+ }
+ }
+ } while (num_duplicates != 0);
+
+ /* Shuffle the random values again */
+ shuffle_input_keys(params);
+
+ /* For testing miss lookup, we insert half and lookup the other half */
+ unsigned int entry_cnt, bf_key_cnt;
+ if (!miss) {
+ entry_cnt = MAX_ENTRIES;
+ bf_key_cnt = KEYS_TO_ADD;
+ } else {
+ entry_cnt = MAX_ENTRIES / 2;
+ bf_key_cnt = KEYS_TO_ADD / 2;
+ }
+ member_params.false_positive_rate = VBF_FALSE_RATE;
+ member_params.key_len = params->key_size;
+ member_params.socket_id = test_socket_id;
+ member_params.num_keys = entry_cnt;
+ member_params.name = "test_member_ht";
+ member_params.iscache = 0;
+ member_params.type = RTE_MEMBER_TYPE_HT;
+ params->setsum[HT] = rte_member_create(&member_params);
+ if (params->setsum[HT] == NULL)
+ fprintf(stderr, "ht create fail\n");
+
+ member_params.name = "test_member_cache";
+ member_params.iscache = 1;
+ params->setsum[CACHE] = rte_member_create(&member_params);
+ if (params->setsum[CACHE] == NULL)
+ fprintf(stderr, "CACHE create fail\n");
+
+ member_params.name = "test_member_vbf";
+ member_params.type = RTE_MEMBER_TYPE_VBF;
+ member_params.num_keys = bf_key_cnt;
+ params->setsum[VBF] = rte_member_create(&member_params);
+ if (params->setsum[VBF] == NULL)
+ fprintf(stderr, "VBF create fail\n");
+ for (i = 0; i < NUM_TYPE; i++) {
+ if (params->setsum[i] == NULL)
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+timed_adds(struct member_perf_params *params, int type)
+{
+ const uint64_t start_tsc = rte_rdtsc();
+ unsigned int i, a;
+ int32_t ret;
+
+ for (i = 0; i < KEYS_TO_ADD; i++) {
+ ret = rte_member_add(params->setsum[type], &keys[i],
+ data[type][i]);
+ if (ret < 0) {
+ printf("Error %d in rte_member_add - key=0x", ret);
+ for (a = 0; a < params->key_size; a++)
+ printf("%02x", keys[i][a]);
+ printf(" value=%d, type: %d\n", data[type][i], type);
+
+ return -1;
+ }
+ }
+
+ const uint64_t end_tsc = rte_rdtsc();
+ const uint64_t time_taken = end_tsc - start_tsc;
+
+ cycles[type][params->cycle][ADD] = time_taken / KEYS_TO_ADD;
+ return 0;
+}
+
+static int
+timed_lookups(struct member_perf_params *params, int type)
+{
+ unsigned int i, j;
+
+ false_data[type][params->cycle] = 0;
+
+ const uint64_t start_tsc = rte_rdtsc();
+ MEMBER_SET_TYPE result;
+ int ret;
+
+ for (i = 0; i < NUM_LOOKUPS / KEYS_TO_ADD; i++) {
+ for (j = 0; j < KEYS_TO_ADD; j++) {
+ ret = rte_member_lookup(params->setsum[type], &keys[j],
+ &result);
+ if (ret < 0) {
+ printf("lookup wrong internally");
+ return -1;
+ }
+ if (result != data[type][j])
+ false_data[type][params->cycle]++;
+ }
+ }
+
+ const uint64_t end_tsc = rte_rdtsc();
+ const uint64_t time_taken = end_tsc - start_tsc;
+
+ cycles[type][params->cycle][LOOKUP] = time_taken / NUM_LOOKUPS;
+
+ return 0;
+}
+
+static int
+timed_lookups_bulk(struct member_perf_params *params, int type)
+{
+ unsigned int i, j, k;
+ MEMBER_SET_TYPE result[BURST_SIZE] = {0};
+ const void *keys_burst[BURST_SIZE];
+ int ret;
+
+ false_data_bulk[type][params->cycle] = 0;
+
+ const uint64_t start_tsc = rte_rdtsc();
+
+ for (i = 0; i < NUM_LOOKUPS / KEYS_TO_ADD; i++) {
+ for (j = 0; j < KEYS_TO_ADD / BURST_SIZE; j++) {
+ for (k = 0; k < BURST_SIZE; k++)
+ keys_burst[k] = keys[j * BURST_SIZE + k];
+
+ ret = rte_member_lookup_bulk(params->setsum[type],
+ &keys_burst[0],
+ BURST_SIZE,
+ result);
+ if (ret <= 0) {
+ printf("lookup bulk has wrong return value\n");
+ return -1;
+ }
+ for (k = 0; k < BURST_SIZE; k++) {
+ uint32_t data_idx = j * BURST_SIZE + k;
+ if (result[k] != data[type][data_idx])
+ false_data_bulk[type][params->cycle]++;
+ }
+ }
+ }
+
+ const uint64_t end_tsc = rte_rdtsc();
+ const uint64_t time_taken = end_tsc - start_tsc;
+
+ cycles[type][params->cycle][LOOKUP_BULK] = time_taken / NUM_LOOKUPS;
+
+ return 0;
+}
+
+
+static int
+timed_lookups_multimatch(struct member_perf_params *params, int type)
+{
+ unsigned int i, j;
+ MEMBER_SET_TYPE result[RTE_MEMBER_BUCKET_ENTRIES] = {0};
+ int ret;
+ false_data_multi[type][params->cycle] = 0;
+
+ const uint64_t start_tsc = rte_rdtsc();
+
+ for (i = 0; i < NUM_LOOKUPS / KEYS_TO_ADD; i++) {
+ for (j = 0; j < KEYS_TO_ADD; j++) {
+ ret = rte_member_lookup_multi(params->setsum[type],
+ &keys[j], RTE_MEMBER_BUCKET_ENTRIES, result);
+ if (type != CACHE && ret <= 0) {
+ printf("lookup multi has wrong return value %d,"
+ "type %d\n", ret, type);
+ }
+ if (result[0] != data[type][j])
+ false_data_multi[type][params->cycle]++;
+ }
+ }
+
+ const uint64_t end_tsc = rte_rdtsc();
+ const uint64_t time_taken = end_tsc - start_tsc;
+
+ cycles[type][params->cycle][LOOKUP_MULTI] = time_taken / NUM_LOOKUPS;
+
+ return 0;
+}
+
+
+static int
+timed_lookups_multimatch_bulk(struct member_perf_params *params, int type)
+{
+ unsigned int i, j, k;
+ MEMBER_SET_TYPE result[BURST_SIZE][RTE_MEMBER_BUCKET_ENTRIES] = {{0} };
+ const void *keys_burst[BURST_SIZE];
+ uint32_t match_count[BURST_SIZE];
+ int ret;
+
+ false_data_multi_bulk[type][params->cycle] = 0;
+
+ const uint64_t start_tsc = rte_rdtsc();
+
+ for (i = 0; i < NUM_LOOKUPS / KEYS_TO_ADD; i++) {
+ for (j = 0; j < KEYS_TO_ADD / BURST_SIZE; j++) {
+ for (k = 0; k < BURST_SIZE; k++)
+ keys_burst[k] = keys[j * BURST_SIZE + k];
+
+ ret = rte_member_lookup_multi_bulk(
+ params->setsum[type],
+ &keys_burst[0], BURST_SIZE,
+ RTE_MEMBER_BUCKET_ENTRIES, match_count,
+ (MEMBER_SET_TYPE *)result);
+ if (ret < 0) {
+ printf("lookup multimatch bulk has wrong return"
+ " value\n");
+ return -1;
+ }
+ for (k = 0; k < BURST_SIZE; k++) {
+ if (type != CACHE && match_count[k] == 0) {
+ printf("lookup multimatch bulk get "
+ "wrong match count\n");
+ return -1;
+ }
+ uint32_t data_idx = j * BURST_SIZE + k;
+ if (result[k][0] != data[type][data_idx])
+ false_data_multi_bulk[type][params->cycle]++;
+ }
+ }
+ }
+
+ const uint64_t end_tsc = rte_rdtsc();
+ const uint64_t time_taken = end_tsc - start_tsc;
+
+ cycles[type][params->cycle][LOOKUP_MULTI_BULK] = time_taken /
+ NUM_LOOKUPS;
+
+ return 0;
+}
+
+static int
+timed_deletes(struct member_perf_params *params, int type)
+{
+ unsigned int i;
+ int32_t ret;
+
+ if (type == VBF)
+ return 0;
+ const uint64_t start_tsc = rte_rdtsc();
+ for (i = 0; i < KEYS_TO_ADD; i++) {
+ ret = rte_member_delete(params->setsum[type], &keys[i],
+ data[type][i]);
+ if (type != CACHE && ret < 0) {
+ printf("delete error\n");
+ return -1;
+ }
+ }
+
+ const uint64_t end_tsc = rte_rdtsc();
+ const uint64_t time_taken = end_tsc - start_tsc;
+
+ cycles[type][params->cycle][DELETE] = time_taken / KEYS_TO_ADD;
+
+ return 0;
+}
+
+
+static int
+timed_miss_lookup(struct member_perf_params *params, int type)
+{
+ unsigned int i, j;
+ int ret;
+
+ false_hit[type][params->cycle] = 0;
+
+ for (i = 0; i < KEYS_TO_ADD / 2; i++) {
+ ret = rte_member_add(params->setsum[type], &keys[i],
+ data[type][i]);
+ if (ret < 0) {
+ unsigned int a;
+ printf("Error %d in rte_member_add - key=0x", ret);
+ for (a = 0; a < params->key_size; a++)
+ printf("%02x", keys[i][a]);
+ printf(" value=%d, type: %d\n", data[type][i], type);
+
+ return -1;
+ }
+ }
+
+ const uint64_t start_tsc = rte_rdtsc();
+ MEMBER_SET_TYPE result;
+
+ for (i = 0; i < 2 * NUM_LOOKUPS / KEYS_TO_ADD; i++) {
+ for (j = KEYS_TO_ADD / 2; j < KEYS_TO_ADD; j++) {
+ ret = rte_member_lookup(params->setsum[type], &keys[j],
+ &result);
+ if (ret < 0) {
+ printf("lookup wrong internally");
+ return -1;
+ }
+ if (result != RTE_MEMBER_NO_MATCH)
+ false_hit[type][params->cycle]++;
+ }
+ }
+
+ const uint64_t end_tsc = rte_rdtsc();
+ const uint64_t time_taken = end_tsc - start_tsc;
+
+ cycles[type][params->cycle][LOOKUP_MISS] = time_taken / NUM_LOOKUPS;
+
+ return 0;
+}
+
+
+
+static void
+perform_frees(struct member_perf_params *params)
+{
+ int i;
+ for (i = 0; i < NUM_TYPE; i++) {
+ if (params->setsum[i] != NULL) {
+ rte_member_free(params->setsum[i]);
+ params->setsum[i] = NULL;
+ }
+ }
+}
+
+static int
+exit_with_fail(const char *testname, struct member_perf_params *params,
+ unsigned int i, unsigned int j)
+{
+ printf("<<<<<Test %s failed at keysize %d iteration %d type %d>>>>>\n",
+ testname, hashtest_key_lens[params->cycle], i, j);
+ perform_frees(params);
+ return -1;
+}
+
+static int
+run_all_tbl_perf_tests(void)
+{
+ unsigned int i, j, k;
+ struct member_perf_params params;
+
+ printf("Measuring performance, please wait\n");
+ fflush(stdout);
+
+ test_socket_id = rte_socket_id();
+
+
+ for (i = 0; i < NUM_KEYSIZES; i++) {
+ if (setup_keys_and_data(&params, i, 0) < 0) {
+ printf("Could not create keys/data/table\n");
+ return -1;
+ }
+ for (j = 0; j < NUM_TYPE; j++) {
+
+ if (timed_adds(&params, j) < 0)
+ return exit_with_fail("timed_adds", &params,
+ i, j);
+
+ for (k = 0; k < NUM_SHUFFLES; k++)
+ shuffle_input_keys(&params);
+
+ if (timed_lookups(&params, j) < 0)
+ return exit_with_fail("timed_lookups", &params,
+ i, j);
+
+ if (timed_lookups_bulk(&params, j) < 0)
+ return exit_with_fail("timed_lookups_bulk",
+ &params, i, j);
+
+ if (timed_lookups_multimatch(&params, j) < 0)
+ return exit_with_fail("timed_lookups_multi",
+ &params, i, j);
+
+ if (timed_lookups_multimatch_bulk(&params, j) < 0)
+ return exit_with_fail("timed_lookups_multi_bulk",
+ &params, i, j);
+
+ if (timed_deletes(&params, j) < 0)
+ return exit_with_fail("timed_deletes", &params,
+ i, j);
+
+ /* Print a dot to show progress on operations */
+ }
+ printf(".");
+ fflush(stdout);
+
+ perform_frees(&params);
+ }
+
+ /* test false postivie rate using un-inserted keys */
+ for (i = 0; i < NUM_KEYSIZES; i++) {
+ if (setup_keys_and_data(&params, i, 1) < 0) {
+ printf("Could not create keys/data/table\n");
+ return -1;
+ }
+ for (j = 0; j < NUM_TYPE; j++) {
+ if (timed_miss_lookup(&params, j) < 0)
+ return exit_with_fail("timed_miss_lookup",
+ &params, i, j);
+ }
+ perform_frees(&params);
+ }
+
+ printf("\nResults (in CPU cycles/operation)\n");
+ printf("-----------------------------------\n");
+ printf("\n%-18s%-18s%-18s%-18s%-18s%-18s%-18s%-18s%-18s\n",
+ "Keysize", "type", "Add", "Lookup", "Lookup_bulk",
+ "lookup_multi", "lookup_multi_bulk", "Delete",
+ "miss_lookup");
+ for (i = 0; i < NUM_KEYSIZES; i++) {
+ for (j = 0; j < NUM_TYPE; j++) {
+ printf("%-18d", hashtest_key_lens[i]);
+ printf("%-18d", j);
+ for (k = 0; k < NUM_OPERATIONS; k++)
+ printf("%-18"PRIu64, cycles[j][i][k]);
+ printf("\n");
+ }
+ }
+
+ printf("\nFalse results rate (and false positive rate)\n");
+ printf("-----------------------------------\n");
+ printf("\n%-18s%-18s%-18s%-18s%-18s%-18s%-18s\n",
+ "Keysize", "type", "fr_single", "fr_bulk", "fr_multi",
+ "fr_multi_bulk", "false_positive_rate");
+ /* key size not influence False rate so just print out one key size */
+ for (i = 0; i < 1; i++) {
+ for (j = 0; j < NUM_TYPE; j++) {
+ printf("%-18d", hashtest_key_lens[i]);
+ printf("%-18d", j);
+ printf("%-18f", (float)false_data[j][i] / NUM_LOOKUPS);
+ printf("%-18f", (float)false_data_bulk[j][i] /
+ NUM_LOOKUPS);
+ printf("%-18f", (float)false_data_multi[j][i] /
+ NUM_LOOKUPS);
+ printf("%-18f", (float)false_data_multi_bulk[j][i] /
+ NUM_LOOKUPS);
+ printf("%-18f", (float)false_hit[j][i] /
+ NUM_LOOKUPS);
+ printf("\n");
+ }
+ }
+
+ return 0;
+}
+
+static int
+test_member_perf(void)
+{
+
+ if (run_all_tbl_perf_tests() < 0)
+ return -1;
+
+ return 0;
+}
+
+REGISTER_TEST_COMMAND(member_perf_autotest, test_member_perf);
--
2.7.4
Yipeng Wang
2017-08-22 00:19:53 UTC
Permalink
This patch adds the documentation for membership library.

Signed-off-by: Yipeng Wang <***@intel.com>
---
doc/api/doxy-api-index.md | 3 +-
doc/api/doxy-api.conf | 1 +
doc/guides/prog_guide/img/member_i1.svg | 1613 +++++++++++++++++++++++++++++++
doc/guides/prog_guide/img/member_i2.svg | 36 +
doc/guides/prog_guide/img/member_i3.svg | 148 +++
doc/guides/prog_guide/img/member_i4.svg | 450 +++++++++
doc/guides/prog_guide/img/member_i5.svg | 163 ++++
doc/guides/prog_guide/img/member_i6.svg | 332 +++++++
doc/guides/prog_guide/img/member_i7.svg | 399 ++++++++
doc/guides/prog_guide/index.rst | 14 +
doc/guides/prog_guide/member_lib.rst | 432 +++++++++
doc/guides/rel_notes/release_17_11.rst | 17 +
12 files changed, 3607 insertions(+), 1 deletion(-)
create mode 100644 doc/guides/prog_guide/img/member_i1.svg
create mode 100644 doc/guides/prog_guide/img/member_i2.svg
create mode 100644 doc/guides/prog_guide/img/member_i3.svg
create mode 100644 doc/guides/prog_guide/img/member_i4.svg
create mode 100644 doc/guides/prog_guide/img/member_i5.svg
create mode 100644 doc/guides/prog_guide/img/member_i6.svg
create mode 100644 doc/guides/prog_guide/img/member_i7.svg
create mode 100644 doc/guides/prog_guide/member_lib.rst

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 19e0d4f..fe87e09 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -105,7 +105,8 @@ The public API headers are grouped by topics:
[LPM IPv4 route] (@ref rte_lpm.h),
[LPM IPv6 route] (@ref rte_lpm6.h),
[ACL] (@ref rte_acl.h),
- [EFD] (@ref rte_efd.h)
+ [EFD] (@ref rte_efd.h),
+ [member] (@ref rte_member.h)

- **QoS**:
[metering] (@ref rte_meter.h),
diff --git a/doc/api/doxy-api.conf b/doc/api/doxy-api.conf
index 823554f..b792d6d 100644
--- a/doc/api/doxy-api.conf
+++ b/doc/api/doxy-api.conf
@@ -58,6 +58,7 @@ INPUT = doc/api/doxy-api-index.md \
lib/librte_mempool \
lib/librte_meter \
lib/librte_metrics \
+ lib/librte_member \
lib/librte_net \
lib/librte_pdump \
lib/librte_pipeline \
diff --git a/doc/guides/prog_guide/img/member_i1.svg b/doc/guides/prog_guide/img/member_i1.svg
new file mode 100644
index 0000000..fc5f56a
--- /dev/null
+++ b/doc/guides/prog_guide/img/member_i1.svg
@@ -0,0 +1,1613 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.0//EN" "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd">
+<!-- Generated by Microsoft Visio 11.0, SVG Export, v1.0 memship_i1.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+ xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="7.18709in" height="4.75757in"
+ viewBox="0 0 517.471 342.545" xml:space="preserve" color-interpolation-filters="sRGB" class="st61">
+ <v:documentProperties v:langID="1033" v:viewMarkup="false">
+ <v:userDefs>
+ <v:ud v:nameU="msvSubprocessMaster" v:prompt="" v:val="VT4(Rectangle)"/>
+ <v:ud v:nameU="msvNoAutoConnect" v:val="VT0(1):26"/>
+ </v:userDefs>
+ </v:documentProperties>
+
+ <style type="text/css">
+ <![CDATA[
+ .st1 {visibility:visible}
+ .st2 {fill:none;stroke:#5b9bd5;stroke-opacity:0.22;stroke-width:3}
+ .st3 {fill:#5b9bd5;font-family:Calibri;font-size:0.666664em;opacity:0.219608}
+ .st4 {font-size:1em}
+ .st5 {fill:none;stroke:#41719c;stroke-width:3}
+ .st6 {fill:#5b9bd5;font-family:Calibri;font-size:0.666664em}
+ .st7 {fill:#5b9bd5;font-family:Calibri;font-size:0.75em;opacity:0.219608}
+ .st8 {fill:#5b9bd5;font-family:Calibri;font-size:0.75em}
+ .st9 {fill:#5b9bd5;fill-opacity:0.22;stroke:#5b9bd5;stroke-opacity:0.22;stroke-width:0.25}
+ .st10 {fill:#5b9bd5;stroke:#c8c8c8;stroke-width:0.25}
+ .st11 {fill:none;stroke:none;stroke-width:0.25}
+ .st12 {fill:#ffffff;font-family:Calibri;font-size:0.499992em;font-weight:bold}
+ .st13 {fill:#ffffff;font-family:Calibri;font-size:0.75em;font-weight:bold}
+ .st14 {marker-end:url(#mrkr5-63);stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st15 {fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-opacity:1;stroke-width:0.28409090909091}
+ .st16 {fill:#5b9bd5;font-family:Calibri;font-size:0.499992em;font-weight:bold}
+ .st17 {stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st18 {fill:#feffff;font-family:Calibri;font-size:0.499992em}
+ .st19 {fill:#deebf6;stroke:#c8c8c8;stroke-width:0.25}
+ .st20 {fill:#000000;font-family:Calibri;font-size:0.499992em}
+ .st21 {marker-end:url(#mrkr5-178);stroke:#ff0000;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st22 {fill:#ff0000;fill-opacity:1;stroke:#ff0000;stroke-opacity:1;stroke-width:0.28409090909091}
+ .st23 {fill:#ff0000;font-family:Calibri;font-size:0.666664em}
+ .st24 {fill:#5b9bd5;fill-opacity:0.22}
+ .st25 {stroke:#5b9bd5;stroke-opacity:0.22;stroke-width:0.25}
+ .st26 {fill:#ffffff}
+ .st27 {stroke:#0070c0;stroke-width:0.25}
+ .st28 {fill:#5b9bd5;stroke:#0070c0;stroke-width:0.25}
+ .st29 {fill:#5b9bd5;stroke:#ffffff;stroke-width:0.25}
+ .st30 {fill:#5b9bd5}
+ .st31 {stroke:#c8c8c8;stroke-width:0.25}
+ .st32 {fill:#acccea;stroke:#c8c8c8;stroke-width:0.25}
+ .st33 {fill:#5b9bd5;fill-opacity:0.22;stroke:none;stroke-linecap:butt;stroke-width:0.75}
+ .st34 {fill:#000000;fill-opacity:0;stroke:none;stroke-linecap:butt;stroke-width:0.75}
+ .st35 {fill:url(#grad30-309);stroke:none;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.75}
+ .st36 {fill:url(#grad25-313);stroke:none;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.75}
+ .st37 {fill:url(#grad35-317);stroke:#308dda;stroke-linecap:butt;stroke-width:0.130208}
+ .st38 {fill:url(#grad36-325);stroke:none;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.75}
+ .st39 {fill:url(#grad40-335);stroke:#000000;stroke-linecap:butt;stroke-width:0.130208}
+ .st40 {fill:url(#grad39-342);stroke:none;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.75}
+ .st41 {fill:url(#grad40-355);stroke:none;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.75}
+ .st42 {fill:none}
+ .st43 {stroke:#308dda;stroke-linecap:butt;stroke-width:0.130208}
+ .st44 {stroke:#ffffff;stroke-linecap:butt;stroke-width:0.130208}
+ .st45 {fill:url(#grad30-383);stroke:none;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.75}
+ .st46 {fill:url(#grad36-396);stroke:none;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.75}
+ .st47 {fill:none;stroke:#c8c8c8;stroke-width:0.75}
+ .st48 {fill:#9a9a9a;stroke:#000000;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.0833333}
+ .st49 {fill:url(#grad40-415);stroke:#000000;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.0833333}
+ .st50 {fill:url(#grad40-419);stroke:#000000;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.0833333}
+ .st51 {stroke:#000000;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25}
+ .st52 {fill:url(#grad35-430);stroke:#000000;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.0833333}
+ .st53 {stroke:#c8c8c8;stroke-width:0.75}
+ .st54 {stroke:#4f88bb;stroke-width:0.75}
+ .st55 {fill:#feffff;font-family:Calibri;font-size:0.416656em}
+ .st56 {fill:#5b9bd5;fill-opacity:0.25;stroke:#5b9bd5;stroke-opacity:0.25;stroke-width:0.75}
+ .st57 {fill:#4f88bb;stroke:#41719c;stroke-width:0.75}
+ .st58 {fill:none;stroke:#5b9bd5;stroke-opacity:0.22;stroke-width:2.25}
+ .st59 {fill:none;stroke:#0070c0;stroke-width:2.25}
+ .st60 {fill:#595959;font-family:Arial;font-size:0.666664em}
+ .st61 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+ ]]>
+ </style>
+
+ <defs id="Patterns_And_Gradients">
+ <linearGradient id="grad30-309" v:fillPattern="30" v:foreground="#97c2e6" v:background="#4274a2" x1="0" y1="1" x2="0"
+ y2="0">
+ <stop offset="0" style="stop-color:#97c2e6;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#4274a2;stop-opacity:1"/>
+ </linearGradient>
+ <linearGradient id="grad25-313" v:fillPattern="25" v:foreground="#5491d3" v:background="#246ba6" x1="0" y1="0" x2="1"
+ y2="0">
+ <stop offset="0" style="stop-color:#5491d3;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#246ba6;stop-opacity:1"/>
+ </linearGradient>
+ <pattern id="grad35-317" v:fillPattern="35" v:foreground="#569bd3" v:background="#aed0ec" x="0" y="0" width="1" height="1"
+ patternContentUnits="objectBoundingBox">
+ <path d="M 0.5 0.5 L 0 0 L 0 1 z" style="fill:url(#grad27-318)"/>
+ <path d="M 0.5 0.5 L 1 0 L 1 1 z" style="fill:url(#grad25-319)"/>
+ <path d="M 0.5 0.5 L 0 0 L 1 0 z" style="fill:url(#grad30-320)"/>
+ <path d="M 0.5 0.5 L 0 1 L 1 1 z" style="fill:url(#grad28-321)"/>
+ </pattern>
+ <linearGradient id="grad27-318" v:fillPattern="35" v:foreground="#569bd3" v:background="#aed0ec" x1="1" y1="0" x2="0"
+ y2="0">
+ <stop offset="0" style="stop-color:#569bd3;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#aed0ec;stop-opacity:1"/>
+ </linearGradient>
+ <linearGradient id="grad25-319" v:fillPattern="35" v:foreground="#569bd3" v:background="#aed0ec" x1="0" y1="0" x2="1"
+ y2="0">
+ <stop offset="0" style="stop-color:#569bd3;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#aed0ec;stop-opacity:1"/>
+ </linearGradient>
+ <linearGradient id="grad30-320" v:fillPattern="35" v:foreground="#569bd3" v:background="#aed0ec" x1="0" y1="1" x2="0"
+ y2="0">
+ <stop offset="0" style="stop-color:#569bd3;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#aed0ec;stop-opacity:1"/>
+ </linearGradient>
+ <linearGradient id="grad28-321" v:fillPattern="35" v:foreground="#569bd3" v:background="#aed0ec" x1="0" y1="0" x2="0"
+ y2="1">
+ <stop offset="0" style="stop-color:#569bd3;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#aed0ec;stop-opacity:1"/>
+ </linearGradient>
+ <radialGradient id="grad36-325" v:fillPattern="36" v:foreground="#c0dff1" v:background="#246ba6" cx="0" cy="0" r="1">
+ <stop offset="0" style="stop-color:#c0dff1;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#246ba6;stop-opacity:1"/>
+ </radialGradient>
+ <radialGradient id="grad40-335" v:fillPattern="40" v:foreground="#c8e5c8" v:background="#19bf19" cx="0.5" cy="0.5" r="0.5">
+ <stop offset="0" style="stop-color:#c8e5c8;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#19bf19;stop-opacity:1"/>
+ </radialGradient>
+ <radialGradient id="grad39-342" v:fillPattern="39" v:foreground="#5599d7" v:background="#b9daf2" cx="1" cy="1" r="1">
+ <stop offset="0" style="stop-color:#5599d7;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#b9daf2;stop-opacity:1"/>
+ </radialGradient>
+ <radialGradient id="grad40-355" v:fillPattern="40" v:foreground="#5599d7" v:background="#214383" cx="0.5" cy="0.5" r="0.5">
+ <stop offset="0" style="stop-color:#5599d7;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#214383;stop-opacity:1"/>
+ </radialGradient>
+ <linearGradient id="grad30-383" v:fillPattern="30" v:foreground="#97c2e6" v:background="#6ba4dc" x1="0" y1="1" x2="0"
+ y2="0">
+ <stop offset="0" style="stop-color:#97c2e6;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#6ba4dc;stop-opacity:1"/>
+ </linearGradient>
+ <radialGradient id="grad36-396" v:fillPattern="36" v:foreground="#89bee9" v:background="#b9daf2" cx="0" cy="0" r="1">
+ <stop offset="0" style="stop-color:#89bee9;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#b9daf2;stop-opacity:1"/>
+ </radialGradient>
+ <radialGradient id="grad40-415" v:fillPattern="40" v:foreground="#000000" v:background="#ffffff" cx="0.5" cy="0.5" r="0.5">
+ <stop offset="0" style="stop-color:#000000;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#ffffff;stop-opacity:1"/>
+ </radialGradient>
+ <radialGradient id="grad40-419" v:fillPattern="40" v:foreground="#ffffff" v:background="#9a9a9a" cx="0.5" cy="0.5" r="0.5">
+ <stop offset="0" style="stop-color:#ffffff;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#9a9a9a;stop-opacity:1"/>
+ </radialGradient>
+ <pattern id="grad35-430" v:fillPattern="35" v:foreground="#ffffff" v:background="#ffcc00" x="0" y="0" width="1" height="1"
+ patternContentUnits="objectBoundingBox">
+ <path d="M 0.5 0.5 L 0 0 L 0 1 z" style="fill:url(#grad27-431)"/>
+ <path d="M 0.5 0.5 L 1 0 L 1 1 z" style="fill:url(#grad25-432)"/>
+ <path d="M 0.5 0.5 L 0 0 L 1 0 z" style="fill:url(#grad30-433)"/>
+ <path d="M 0.5 0.5 L 0 1 L 1 1 z" style="fill:url(#grad28-434)"/>
+ </pattern>
+ <linearGradient id="grad27-431" v:fillPattern="35" v:foreground="#ffffff" v:background="#ffcc00" x1="1" y1="0" x2="0"
+ y2="0">
+ <stop offset="0" style="stop-color:#ffffff;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#ffcc00;stop-opacity:1"/>
+ </linearGradient>
+ <linearGradient id="grad25-432" v:fillPattern="35" v:foreground="#ffffff" v:background="#ffcc00" x1="0" y1="0" x2="1"
+ y2="0">
+ <stop offset="0" style="stop-color:#ffffff;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#ffcc00;stop-opacity:1"/>
+ </linearGradient>
+ <linearGradient id="grad30-433" v:fillPattern="35" v:foreground="#ffffff" v:background="#ffcc00" x1="0" y1="1" x2="0"
+ y2="0">
+ <stop offset="0" style="stop-color:#ffffff;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#ffcc00;stop-opacity:1"/>
+ </linearGradient>
+ <linearGradient id="grad28-434" v:fillPattern="35" v:foreground="#ffffff" v:background="#ffcc00" x1="0" y1="0" x2="0"
+ y2="1">
+ <stop offset="0" style="stop-color:#ffffff;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#ffcc00;stop-opacity:1"/>
+ </linearGradient>
+ </defs>
+ <defs id="Markers">
+ <g id="lend5">
+ <path d="M 2 1 L 0 0 L 1.98117 -0.993387 C 1.67173 -0.364515 1.67301 0.372641 1.98465 1.00043 " style="stroke:none"/>
+ </g>
+ <marker id="mrkr5-63" class="st15" v:arrowType="5" v:arrowSize="2" v:setback="6.16" refX="-6.16" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-3.52,-3.52) "/>
+ </marker>
+ <marker id="mrkr5-178" class="st22" v:arrowType="5" v:arrowSize="2" v:setback="5.8" refX="-5.8" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-3.52,-3.52) "/>
+ </marker>
+ </defs>
+ <g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeOrder" v:val="VT0(0):26"/>
+ </v:userDefs>
+ <title>Page-1</title>
+ <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+ <v:layer v:name="Flowchart" v:index="0"/>
+ <g id="group165-1" transform="translate(21.7794,-24.0978)" v:mID="165" v:groupContext="group">
+ <title>Sheet.165</title>
+ <g id="group1-2" transform="translate(308.647,-25.7109)" v:mID="1" v:groupContext="group">
+ <title>Sheet.1</title>
+ <g id="shape2-3" v:mID="2" v:groupContext="shape" transform="translate(11.5732,-58.1913)">
+ <title>Circle</title>
+ <desc>List 1 matching Criteria 1</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="27.2233" cy="315.322" width="47.65" height="40.835"/>
+ <g id="shadow2-4" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 315.32 A27.2233 27.2233 0 1 1 54.45 315.32 A27.2233 27.2233 0 1 1 0 315.32 Z" class="st2"/>
+ <text x="18.79" y="308.12" class="st3" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>List 1 <tspan
+ x="12.08" dy="1.2em" class="st4">matching </tspan><tspan x="12.29" dy="1.2em" class="st4">Criteria </tspan>1</text> </g>
+ <path d="M0 315.32 A27.2233 27.2233 0 1 1 54.45 315.32 A27.2233 27.2233 0 1 1 0 315.32 Z" class="st5"/>
+ <text x="18.79" y="308.12" class="st6" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>List 1 <tspan
+ x="12.08" dy="1.2em" class="st4">matching </tspan><tspan x="12.29" dy="1.2em" class="st4">Criteria </tspan>1</text> </g>
+ <g id="shape3-13" v:mID="3" v:groupContext="shape" transform="translate(58.9839,-58.9839)">
+ <title>Circle.23</title>
+ <desc>List 2</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="27.2233" cy="315.322" width="47.65" height="40.835"/>
+ <g id="shadow3-14" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 315.32 A27.2233 27.2233 0 1 1 54.45 315.32 A27.2233 27.2233 0 1 1 0 315.32 Z" class="st2"/>
+ <text x="17.73" y="318.02" class="st7" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>List 2</text> </g>
+ <path d="M0 315.32 A27.2233 27.2233 0 1 1 54.45 315.32 A27.2233 27.2233 0 1 1 0 315.32 Z" class="st5"/>
+ <text x="17.73" y="318.02" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>List 2</text> </g>
+ <g id="shape4-19" v:mID="4" v:groupContext="shape">
+ <title>Circle.24</title>
+ <desc>List 1 matching Criteria 1</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="27.2233" cy="315.322" width="47.65" height="40.835"/>
+ <g id="shadow4-20" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 315.32 A27.2233 27.2233 0 1 1 54.45 315.32 A27.2233 27.2233 0 1 1 0 315.32 Z" class="st2"/>
+ <text x="18.79" y="308.12" class="st3" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>List 1 <tspan
+ x="12.08" dy="1.2em" class="st4">matching </tspan><tspan x="12.29" dy="1.2em" class="st4">Criteria </tspan>1</text> </g>
+ <path d="M0 315.32 A27.2233 27.2233 0 1 1 54.45 315.32 A27.2233 27.2233 0 1 1 0 315.32 Z" class="st5"/>
+ <text x="18.79" y="308.12" class="st6" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>List 1 <tspan
+ x="12.08" dy="1.2em" class="st4">matching </tspan><tspan x="12.29" dy="1.2em" class="st4">Criteria </tspan>1</text> </g>
+ <g id="group5-29" transform="translate(50.7413,-4.53722)" v:mID="5" v:groupContext="group">
+ <title>Sheet.5</title>
+ <g id="shape6-30" v:mID="6" v:groupContext="shape" transform="translate(344.2,300.5) rotate(90)">
+ <title>Triangle</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow6-31" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,1.9728,-0.3456)" class="st1">
+ <path d="M42.04 342.55 L21.02 318.64 L0 342.55 L42.04 342.55 Z" class="st9"/>
+ </g>
+ <path d="M42.04 342.55 L21.02 318.64 L0 342.55 L42.04 342.55 Z" class="st10"/>
+ </g>
+ <g id="shape7-34" v:mID="7" v:groupContext="shape" transform="translate(-0.884982,-14.7157)">
+ <title>Sheet.7</title>
+ <desc>setsum</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="12.9268" cy="336.238" width="25.86" height="12.6135"/>
+ <rect x="0" y="329.932" width="25.8535" height="12.6135" class="st11"/>
+ <text x="6.37" y="334.44" class="st12" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>setsu<tspan
+ x="10.49" dy="1.2em" class="st4">m</tspan></text> </g>
+ </g>
+ <g id="shape8-38" v:mID="8" v:groupContext="shape" transform="translate(72.5955,0)">
+ <title>Circle.29</title>
+ <desc>List 2 matching Criteria 2</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="27.2233" cy="315.322" width="47.65" height="40.835"/>
+ <g id="shadow8-39" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 315.32 A27.2233 27.2233 0 1 1 54.45 315.32 A27.2233 27.2233 0 1 1 0 315.32 Z" class="st2"/>
+ <text x="18.79" y="308.12" class="st3" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>List 2 <tspan
+ x="12.08" dy="1.2em" class="st4">matching </tspan><tspan x="12.29" dy="1.2em" class="st4">Criteria </tspan>2</text> </g>
+ <path d="M0 315.32 A27.2233 27.2233 0 1 1 54.45 315.32 A27.2233 27.2233 0 1 1 0 315.32 Z" class="st5"/>
+ <text x="18.79" y="308.12" class="st6" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>List 2 <tspan
+ x="12.08" dy="1.2em" class="st4">matching </tspan><tspan x="12.29" dy="1.2em" class="st4">Criteria </tspan>2</text> </g>
+ </g>
+ <g id="group9-48" transform="translate(31.6515,-49.9094)" v:mID="9" v:groupContext="group">
+ <title>Sheet.9</title>
+ <g id="group10-49" transform="translate(99.5691,0)" v:mID="10" v:groupContext="group">
+ <title>Sheet.10</title>
+ <g id="shape11-50" v:mID="11" v:groupContext="shape" transform="translate(346.175,275.999) rotate(90)">
+ <title>Triangle</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow11-51" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,1.9728,-0.3456)" class="st1">
+ <path d="M66.55 342.55 L33.27 290.12 L0 342.55 L66.55 342.55 Z" class="st9"/>
+ </g>
+ <path d="M66.55 342.55 L33.27 290.12 L0 342.55 L66.55 342.55 Z" class="st10"/>
+ </g>
+ <g id="shape12-54" v:mID="12" v:groupContext="shape" transform="translate(355.063,285.074) rotate(90)">
+ <title>Sheet.12</title>
+ <desc>Set Summary</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="24.1985" cy="332.563" width="48.4" height="19.9638"/>
+ <rect x="0" y="322.581" width="48.397" height="19.9638" class="st11"/>
+ <text x="18.25" y="329.86" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Set <tspan
+ x="6.38" dy="1.2em" class="st4">Summary</tspan></text> </g>
+ </g>
+ <g id="shape13-58" v:mID="13" v:groupContext="shape" transform="translate(57.5835,-54.4467)">
+ <title>Sheet.13</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L38.9 342.55" class="st14"/>
+ </g>
+ <g id="shape14-64" v:mID="14" v:groupContext="shape" transform="translate(20.2363,-51.8439)">
+ <title>Sheet.14</title>
+ <desc>Flow Key</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="25.3328" cy="333.471" width="50.67" height="18.1489"/>
+ <rect x="0" y="324.396" width="50.6656" height="18.1489" class="st11"/>
+ <text x="14.12" y="335.27" class="st16" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow Key</text> </g>
+ <g id="shape15-67" v:mID="15" v:groupContext="shape" transform="translate(5.02911,1.60865) rotate(-26.0815)">
+ <title>Sheet.15</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L39.25 342.55" class="st14"/>
+ </g>
+ <g id="shape16-72" v:mID="16" v:groupContext="shape" transform="translate(155.629,-33.273)">
+ <title>Sheet.16</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L38.34 342.55" class="st14"/>
+ </g>
+ <g id="shape17-77" v:mID="17" v:groupContext="shape" transform="translate(304.141,0.595416) rotate(25.6934)">
+ <title>Sheet.17</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L42.68 342.55" class="st14"/>
+ </g>
+ <g id="shape18-82" v:mID="18" v:groupContext="shape" transform="translate(102.642,654.842) rotate(180)">
+ <title>Sheet.18</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L30.14 342.55" class="st14"/>
+ </g>
+ <g id="shape19-87" v:mID="19" v:groupContext="shape" transform="translate(-15.1809,-33.9928)">
+ <title>Sheet.19</title>
+ <desc>New Flow =&#62; New Assignment</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="42.75" cy="338.045" width="85.5" height="9"/>
+ <rect x="0" y="333.545" width="85.5" height="9" class="st11"/>
+ <text x="5.06" y="339.85" class="st16" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>New Flow =&#62; New Assignment</text> </g>
+ <g id="shape20-90" v:mID="20" v:groupContext="shape" transform="translate(102.844,679.041) rotate(180)">
+ <title>Sheet.20</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L30.14 342.55" class="st14"/>
+ </g>
+ <g id="shape21-95" v:mID="21" v:groupContext="shape" transform="translate(-35.4309,-11.4928)">
+ <title>Sheet.21</title>
+ <desc>Old Flow =&#62; forward to specific thread</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="337.971" width="108" height="9.14889"/>
+ <rect x="0" y="333.396" width="108" height="9.14889" class="st11"/>
+ <text x="6.36" y="339.77" class="st16" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Old Flow =&#62; forward to specific thread</text> </g>
+ <g id="shape22-98" v:mID="22" v:groupContext="shape" transform="translate(541.496,275.999) rotate(90)">
+ <title>Sheet.22</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 335.81 C2.14 344.21 5.09 343.6 7.56 340.31 C10.62 336.25 12.94 328.1 18.15 335.81" class="st17"/>
+ </g>
+ <g id="shape23-101" v:mID="23" v:groupContext="shape" transform="translate(541.496,300.198) rotate(90)">
+ <title>Sheet.23</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 335.81 C2.14 344.21 5.09 343.6 7.56 340.31 C10.62 336.25 12.94 328.1 18.15 335.81" class="st17"/>
+ </g>
+ <g id="shape24-104" v:mID="24" v:groupContext="shape" transform="translate(541.496,324.396) rotate(90)">
+ <title>Sheet.24</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 335.81 C2.14 344.21 5.09 343.6 7.56 340.31 C10.62 336.25 12.94 328.1 18.15 335.81" class="st17"/>
+ </g>
+ </g>
+ <g id="group25-107" transform="translate(285.961,-178.628)" v:mID="25" v:groupContext="group">
+ <title>Sheet.25</title>
+ <g id="shape26-108" v:mID="26" v:groupContext="shape" transform="translate(51.2583,-51.2583)">
+ <title>Circle</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow26-109" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st9"/>
+ </g>
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st10"/>
+ </g>
+ <g id="shape27-112" v:mID="27" v:groupContext="shape" transform="translate(107.177,-55.9182)">
+ <title>Circle.156</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow27-113" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st9"/>
+ </g>
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st10"/>
+ </g>
+ <g id="shape28-116" v:mID="28" v:groupContext="shape" transform="translate(79.2174,-83.8773)">
+ <title>Circle.157</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow28-117" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st9"/>
+ </g>
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st10"/>
+ </g>
+ <g id="shape29-120" v:mID="29" v:groupContext="shape" transform="translate(153.775,-51.2583)">
+ <title>Circle.158</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow29-121" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st9"/>
+ </g>
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st10"/>
+ </g>
+ <g id="shape30-124" v:mID="30" v:groupContext="shape" transform="translate(93.197,-18.6394)">
+ <title>Circle.159</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow30-125" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st9"/>
+ </g>
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st10"/>
+ </g>
+ <g id="shape31-128" v:mID="31" v:groupContext="shape" transform="translate(27.4102,-57.9329) rotate(-7.12502)">
+ <title>Sheet.31</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L31.41 342.55" class="st14"/>
+ </g>
+ <g id="shape32-133" v:mID="32" v:groupContext="shape" transform="translate(182.13,-60.5772) rotate(9.46232)">
+ <title>Sheet.32</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L22.18 342.55" class="st14"/>
+ </g>
+ <g id="shape33-138" v:mID="33" v:groupContext="shape" transform="translate(47.8843,595.237) rotate(-160.346)">
+ <title>Sheet.33</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L63.11 342.55" class="st14"/>
+ </g>
+ <g id="shape34-143" v:mID="34" v:groupContext="shape" transform="translate(292.945,525.785) rotate(141.977)">
+ <title>Sheet.34</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L20.97 342.55" class="st14"/>
+ </g>
+ <g id="shape35-148" v:mID="35" v:groupContext="shape" transform="translate(-95.8971,591.793) rotate(-145.945)">
+ <title>Sheet.35</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L28.55 342.55" class="st14"/>
+ </g>
+ <g id="shape36-153" v:mID="36" v:groupContext="shape" transform="translate(37.2788,2.27374E-013)">
+ <title>Rectangle.167</title>
+ <desc>SUM</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="10.8652" cy="335.555" width="21.74" height="13.9795"/>
+ <g id="shadow36-154" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="328.566" width="21.7305" height="13.9795" class="st9"/>
+ </g>
+ <rect x="0" y="328.566" width="21.7305" height="13.9795" class="st10"/>
+ <text x="5" y="337.36" class="st18" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>SUM</text> </g>
+ <g id="shape37-158" v:mID="37" v:groupContext="shape" transform="translate(55.9182,2.27374E-013)">
+ <title>Rectangle.168</title>
+ <desc>Packet</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="23.2992" cy="335.555" width="46.6" height="13.9795"/>
+ <g id="shadow37-159" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="328.566" width="46.5985" height="13.9795" class="st9"/>
+ </g>
+ <rect x="0" y="328.566" width="46.5985" height="13.9795" class="st19"/>
+ <text x="15.18" y="337.36" class="st20" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Packet</text> </g>
+ <g id="shape38-163" v:mID="38" v:groupContext="shape" transform="translate(-1.65867E-013,-32.6189)">
+ <title>Rectangle.169</title>
+ <desc>SUM</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="10.3796" cy="335.555" width="20.76" height="13.9795"/>
+ <g id="shadow38-164" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="328.566" width="20.7593" height="13.9795" class="st9"/>
+ </g>
+ <rect x="0" y="328.566" width="20.7593" height="13.9795" class="st10"/>
+ <text x="4.51" y="337.36" class="st18" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>SUM</text> </g>
+ <g id="shape39-168" v:mID="39" v:groupContext="shape" transform="translate(18.6394,-32.6189)">
+ <title>Rectangle.170</title>
+ <desc>Packet</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="23.2992" cy="335.555" width="46.6" height="13.9795"/>
+ <g id="shadow39-169" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="328.566" width="46.5985" height="13.9795" class="st9"/>
+ </g>
+ <rect x="0" y="328.566" width="46.5985" height="13.9795" class="st19"/>
+ <text x="15.18" y="337.36" class="st20" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Packet</text> </g>
+ <g id="shape40-173" v:mID="40" v:groupContext="shape" transform="translate(197.019,626.053) rotate(161.565)">
+ <title>Sheet.40</title>
+ <path d="M0 328.31 A55.7483 27.2427 -124.2 0 0 42.37 334.19 L42.47 333.85" class="st21"/>
+ </g>
+ <g id="shape41-179" v:mID="41" v:groupContext="shape" transform="translate(154.607,584.177) rotate(161.121)">
+ <title>Sheet.41</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 319.39 A80.5593 29.9756 -101.99 0 0 41.7 325.37 L41.79 325.02" class="st21"/>
+ </g>
+ <g id="shape42-184" v:mID="42" v:groupContext="shape" transform="translate(3.02481,-66.7025)">
+ <title>Sheet.42</title>
+ <desc>Encode ID</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="19.4569" cy="335.555" width="38.92" height="13.9795"/>
+ <rect x="0" y="328.566" width="38.9138" height="13.9795" class="st11"/>
+ <text x="7.51" y="333.16" class="st23" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Encode <tspan
+ x="15.99" dy="1.2em" class="st4">ID</tspan></text> </g>
+ </g>
+ <g id="group43-188" transform="translate(12.0993,-165.858)" v:mID="43" v:groupContext="group">
+ <title>Sheet.43</title>
+ <g id="group44-189" transform="translate(7.21495,-75.757)" v:mID="44" v:groupContext="group" v:layerMember="0">
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ <v:ud v:nameU="ConnGap" v:prompt="" v:val="VT0(0.083333333333333):0"/>
+ </v:userDefs>
+ <title>User</title>
+ <g id="shape45-190" v:mID="45" v:groupContext="shape" v:layerMember="0"
+ transform="translate(13.3353,-1.13687E-013)">
+ <title>Sheet.45</title>
+ <g id="shadow45-191" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42 ZM20.96
+ 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52
+ 13.66 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55
+ L21.12 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36
+ 329.46 C22.2 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77 Z"
+ class="st24"/>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42"
+ class="st25"/>
+ <path d="M20.96 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52
+ 13.66 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55
+ L21.12 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36
+ 329.46 C22.2 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77"
+ class="st25"/>
+ <path d="M18.1 342.55 L18.1 338.37" class="st25"/>
+ <path d="M15.03 337.32 L15.03 333.71" class="st25"/>
+ <path d="M21.12 337.32 L21.12 333.71" class="st25"/>
+ <path d="M0 337.32 L13.47 337.32" class="st25"/>
+ </g>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42 ZM20.96
+ 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52 13.66
+ 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55 L21.12
+ 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36 329.46 C22.2
+ 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77 Z" class="st26"/>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42"
+ class="st27"/>
+ <path d="M20.96 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52 13.66
+ 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55 L21.12
+ 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36 329.46 C22.2
+ 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77" class="st27"/>
+ <path d="M18.1 342.55 L18.1 338.37" class="st27"/>
+ <path d="M15.03 337.32 L15.03 333.71" class="st27"/>
+ <path d="M21.12 337.32 L21.12 333.71" class="st27"/>
+ <path d="M0 337.32 L13.47 337.32" class="st27"/>
+ </g>
+ <g id="shape46-206" v:mID="46" v:groupContext="shape" v:layerMember="0" transform="translate(0,-8.39743)">
+ <title>Sheet.46</title>
+ <g id="shadow46-207" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M21.09 325.52 C21.09 325.13 20.96 324.79 20.74 324.51 C20.59 324.32 20.4 324.16 20.19 324.06
+ C19.97 323.95 19.72 323.89 19.46 323.89 L3.55 323.89 C3.16 323.89 2.82 324.02 2.54 324.23
+ C2.35 324.38 2.19 324.57 2.09 324.79 C1.98 325.01 1.92 325.25 1.92 325.52 L1.92 336.04 L21.09
+ 336.04 L21.09 325.52 ZM21.18 337.33 L1.77 337.33 L0 340.51 L0 342.55 L23.06 342.55 L23.06
+ 340.51 L21.18 337.33 Z" class="st9"/>
+ </g>
+ <path d="M21.09 325.52 C21.09 325.13 20.96 324.79 20.74 324.51 C20.59 324.32 20.4 324.16 20.19 324.06 C19.97
+ 323.95 19.72 323.89 19.46 323.89 L3.55 323.89 C3.16 323.89 2.82 324.02 2.54 324.23 C2.35 324.38
+ 2.19 324.57 2.09 324.79 C1.98 325.01 1.92 325.25 1.92 325.52 L1.92 336.04 L21.09 336.04 L21.09
+ 325.52 ZM21.18 337.33 L1.77 337.33 L0 340.51 L0 342.55 L23.06 342.55 L23.06 340.51 L21.18 337.33
+ Z" class="st28"/>
+ </g>
+ <g id="shape47-210" v:mID="47" v:groupContext="shape" v:layerMember="0" transform="translate(3.19243,-16.175)">
+ <title>Sheet.47</title>
+ <v:userDefs>
+ <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(#5b9bd5)"/>
+ </v:userDefs>
+ <path d="M16.62 342.55 L16.62 333.29 C16.62 333.23 16.61 333.18 16.58 333.13 C16.55 333.07 16.5 333.02 16.44
+ 332.98 C16.39 332.95 16.33 332.94 16.27 332.94 L0.35 332.94 C0.29 332.94 0.24 332.95 0.19 332.98
+ C0.13 333.01 0.08 333.07 0.04 333.12 C0.02 333.17 0 333.23 0 333.29 L0 342.55 L16.62 342.55
+ Z" class="st29"/>
+ </g>
+ <g id="shape48-212" v:mID="48" v:groupContext="shape" v:layerMember="0" transform="translate(1.97942,-10.81)">
+ <title>Sheet.48</title>
+ <v:userDefs>
+ <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(#5b9bd5)"/>
+ </v:userDefs>
+ <path d="M0.96 340.83 L0 342.55 L19.06 342.55 L18.1 340.83 L0.96 340.83 Z" class="st26"/>
+ </g>
+ </g>
+ <g id="group49-215" transform="translate(7.21495,-47.1858)" v:mID="49" v:groupContext="group" v:layerMember="0">
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ <v:ud v:nameU="ConnGap" v:prompt="" v:val="VT0(0.083333333333333):0"/>
+ </v:userDefs>
+ <title>User.7</title>
+ <g id="shape50-216" v:mID="50" v:groupContext="shape" v:layerMember="0"
+ transform="translate(13.3353,-1.13687E-013)">
+ <title>Sheet.50</title>
+ <g id="shadow50-217" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42 ZM20.96
+ 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52
+ 13.66 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55
+ L21.12 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36
+ 329.46 C22.2 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77 Z"
+ class="st24"/>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42"
+ class="st25"/>
+ <path d="M20.96 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52
+ 13.66 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55
+ L21.12 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36
+ 329.46 C22.2 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77"
+ class="st25"/>
+ <path d="M18.1 342.55 L18.1 338.37" class="st25"/>
+ <path d="M15.03 337.32 L15.03 333.71" class="st25"/>
+ <path d="M21.12 337.32 L21.12 333.71" class="st25"/>
+ <path d="M0 337.32 L13.47 337.32" class="st25"/>
+ </g>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42 ZM20.96
+ 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52 13.66
+ 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55 L21.12
+ 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36 329.46 C22.2
+ 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77 Z" class="st26"/>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42"
+ class="st27"/>
+ <path d="M20.96 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52 13.66
+ 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55 L21.12
+ 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36 329.46 C22.2
+ 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77" class="st27"/>
+ <path d="M18.1 342.55 L18.1 338.37" class="st27"/>
+ <path d="M15.03 337.32 L15.03 333.71" class="st27"/>
+ <path d="M21.12 337.32 L21.12 333.71" class="st27"/>
+ <path d="M0 337.32 L13.47 337.32" class="st27"/>
+ </g>
+ <g id="shape51-232" v:mID="51" v:groupContext="shape" v:layerMember="0" transform="translate(0,-8.39743)">
+ <title>Sheet.51</title>
+ <g id="shadow51-233" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M21.09 325.52 C21.09 325.13 20.96 324.79 20.74 324.51 C20.59 324.32 20.4 324.16 20.19 324.06
+ C19.97 323.95 19.72 323.89 19.46 323.89 L3.55 323.89 C3.16 323.89 2.82 324.02 2.54 324.23
+ C2.35 324.38 2.19 324.57 2.09 324.79 C1.98 325.01 1.92 325.25 1.92 325.52 L1.92 336.04 L21.09
+ 336.04 L21.09 325.52 ZM21.18 337.33 L1.77 337.33 L0 340.51 L0 342.55 L23.06 342.55 L23.06
+ 340.51 L21.18 337.33 Z" class="st9"/>
+ </g>
+ <path d="M21.09 325.52 C21.09 325.13 20.96 324.79 20.74 324.51 C20.59 324.32 20.4 324.16 20.19 324.06 C19.97
+ 323.95 19.72 323.89 19.46 323.89 L3.55 323.89 C3.16 323.89 2.82 324.02 2.54 324.23 C2.35 324.38
+ 2.19 324.57 2.09 324.79 C1.98 325.01 1.92 325.25 1.92 325.52 L1.92 336.04 L21.09 336.04 L21.09
+ 325.52 ZM21.18 337.33 L1.77 337.33 L0 340.51 L0 342.55 L23.06 342.55 L23.06 340.51 L21.18 337.33
+ Z" class="st28"/>
+ </g>
+ <g id="shape52-236" v:mID="52" v:groupContext="shape" v:layerMember="0" transform="translate(3.19243,-16.175)">
+ <title>Sheet.52</title>
+ <v:userDefs>
+ <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(#5b9bd5)"/>
+ </v:userDefs>
+ <path d="M16.62 342.55 L16.62 333.29 C16.62 333.23 16.61 333.18 16.58 333.13 C16.55 333.07 16.5 333.02 16.44
+ 332.98 C16.39 332.95 16.33 332.94 16.27 332.94 L0.35 332.94 C0.29 332.94 0.24 332.95 0.19 332.98
+ C0.13 333.01 0.08 333.07 0.04 333.12 C0.02 333.17 0 333.23 0 333.29 L0 342.55 L16.62 342.55
+ Z" class="st29"/>
+ </g>
+ <g id="shape53-238" v:mID="53" v:groupContext="shape" v:layerMember="0" transform="translate(1.97942,-10.81)">
+ <title>Sheet.53</title>
+ <v:userDefs>
+ <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(#5b9bd5)"/>
+ </v:userDefs>
+ <path d="M0.96 340.83 L0 342.55 L19.06 342.55 L18.1 340.83 L0.96 340.83 Z" class="st26"/>
+ </g>
+ </g>
+ <g id="group54-241" transform="translate(7.21495,-18.6146)" v:mID="54" v:groupContext="group" v:layerMember="0">
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ <v:ud v:nameU="ConnGap" v:prompt="" v:val="VT0(0.083333333333333):0"/>
+ </v:userDefs>
+ <title>User.12</title>
+ <g id="shape55-242" v:mID="55" v:groupContext="shape" v:layerMember="0"
+ transform="translate(13.3353,-1.13687E-013)">
+ <title>Sheet.55</title>
+ <g id="shadow55-243" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42 ZM20.96
+ 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52
+ 13.66 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55
+ L21.12 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36
+ 329.46 C22.2 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77 Z"
+ class="st24"/>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42"
+ class="st25"/>
+ <path d="M20.96 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52
+ 13.66 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55
+ L21.12 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36
+ 329.46 C22.2 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77"
+ class="st25"/>
+ <path d="M18.1 342.55 L18.1 338.37" class="st25"/>
+ <path d="M15.03 337.32 L15.03 333.71" class="st25"/>
+ <path d="M21.12 337.32 L21.12 333.71" class="st25"/>
+ <path d="M0 337.32 L13.47 337.32" class="st25"/>
+ </g>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42 ZM20.96
+ 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52 13.66
+ 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55 L21.12
+ 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36 329.46 C22.2
+ 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77 Z" class="st26"/>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42"
+ class="st27"/>
+ <path d="M20.96 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52 13.66
+ 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55 L21.12
+ 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36 329.46 C22.2
+ 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77" class="st27"/>
+ <path d="M18.1 342.55 L18.1 338.37" class="st27"/>
+ <path d="M15.03 337.32 L15.03 333.71" class="st27"/>
+ <path d="M21.12 337.32 L21.12 333.71" class="st27"/>
+ <path d="M0 337.32 L13.47 337.32" class="st27"/>
+ </g>
+ <g id="shape56-258" v:mID="56" v:groupContext="shape" v:layerMember="0" transform="translate(0,-8.39743)">
+ <title>Sheet.56</title>
+ <g id="shadow56-259" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M21.09 325.52 C21.09 325.13 20.96 324.79 20.74 324.51 C20.59 324.32 20.4 324.16 20.19 324.06
+ C19.97 323.95 19.72 323.89 19.46 323.89 L3.55 323.89 C3.16 323.89 2.82 324.02 2.54 324.23
+ C2.35 324.38 2.19 324.57 2.09 324.79 C1.98 325.01 1.92 325.25 1.92 325.52 L1.92 336.04 L21.09
+ 336.04 L21.09 325.52 ZM21.18 337.33 L1.77 337.33 L0 340.51 L0 342.55 L23.06 342.55 L23.06
+ 340.51 L21.18 337.33 Z" class="st9"/>
+ </g>
+ <path d="M21.09 325.52 C21.09 325.13 20.96 324.79 20.74 324.51 C20.59 324.32 20.4 324.16 20.19 324.06 C19.97
+ 323.95 19.72 323.89 19.46 323.89 L3.55 323.89 C3.16 323.89 2.82 324.02 2.54 324.23 C2.35 324.38
+ 2.19 324.57 2.09 324.79 C1.98 325.01 1.92 325.25 1.92 325.52 L1.92 336.04 L21.09 336.04 L21.09
+ 325.52 ZM21.18 337.33 L1.77 337.33 L0 340.51 L0 342.55 L23.06 342.55 L23.06 340.51 L21.18 337.33
+ Z" class="st28"/>
+ </g>
+ <g id="shape57-262" v:mID="57" v:groupContext="shape" v:layerMember="0" transform="translate(3.19243,-16.175)">
+ <title>Sheet.57</title>
+ <v:userDefs>
+ <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(#5b9bd5)"/>
+ </v:userDefs>
+ <path d="M16.62 342.55 L16.62 333.29 C16.62 333.23 16.61 333.18 16.58 333.13 C16.55 333.07 16.5 333.02 16.44
+ 332.98 C16.39 332.95 16.33 332.94 16.27 332.94 L0.35 332.94 C0.29 332.94 0.24 332.95 0.19 332.98
+ C0.13 333.01 0.08 333.07 0.04 333.12 C0.02 333.17 0 333.23 0 333.29 L0 342.55 L16.62 342.55
+ Z" class="st29"/>
+ </g>
+ <g id="shape58-264" v:mID="58" v:groupContext="shape" v:layerMember="0" transform="translate(1.97942,-10.81)">
+ <title>Sheet.58</title>
+ <v:userDefs>
+ <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(#5b9bd5)"/>
+ </v:userDefs>
+ <path d="M0.96 340.83 L0 342.55 L19.06 342.55 L18.1 340.83 L0.96 340.83 Z" class="st26"/>
+ </g>
+ </g>
+ <g id="group59-267" transform="translate(171.161,-45.6707)" v:mID="59" v:groupContext="group" v:layerMember="0">
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ <v:ud v:nameU="ConnGap" v:prompt="" v:val="VT0(0.083333333333333):0"/>
+ </v:userDefs>
+ <title>Data Center</title>
+ <g id="shape60-268" v:mID="60" v:groupContext="shape" v:layerMember="0">
+ <title>Sheet.60</title>
+ <g id="shadow60-269" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <ellipse cx="37.8785" cy="331.299" rx="37.8785" ry="11.246" class="st9"/>
+ </g>
+ <ellipse cx="37.8785" cy="331.299" rx="37.8785" ry="11.246" class="st10"/>
+ </g>
+ <g id="shape61-272" v:mID="61" v:groupContext="shape" v:layerMember="0" transform="translate(6.86487,-7.30475)">
+ <title>Sheet.61</title>
+ <g id="shadow61-273" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M54.1 311.79 L43.28 311.79 L43.28 342.55 L62.03 342.55 L62.03 311.79 L54.1 311.79 ZM43.28 332.44
+ L43.28 311.79 L51.21 311.79 L51.21 301.69 L32.33 301.69 L32.33 311.79 L40.39 311.79 L40.39
+ 332.44 L43.28 332.44 ZM40.39 301.69 L40.39 293.03 L21.64 293.03 L21.64 301.69 L29.57 301.69
+ L29.57 311.79 L32.46 311.79 L32.46 301.69 L40.39 301.69 ZM32.46 311.79 L21.64 311.79 L21.64
+ 342.55 L40.39 342.55 L40.39 311.79 L32.46 311.79 ZM10.82 311.79 L0 311.79 L0 342.55 L18.75
+ 342.55 L18.75 311.79 L10.82 311.79 ZM21.64 311.79 L29.57 311.79 L29.57 301.69 L10.82 301.69
+ L10.82 311.79 L18.75 311.79 L18.75 332.44 L21.64 332.44 L21.64 311.79 Z" class="st9"/>
+ </g>
+ <path d="M54.1 311.79 L43.28 311.79 L43.28 342.55 L62.03 342.55 L62.03 311.79 L54.1 311.79 ZM43.28 332.44
+ L43.28 311.79 L51.21 311.79 L51.21 301.69 L32.33 301.69 L32.33 311.79 L40.39 311.79 L40.39 332.44
+ L43.28 332.44 ZM40.39 301.69 L40.39 293.03 L21.64 293.03 L21.64 301.69 L29.57 301.69 L29.57
+ 311.79 L32.46 311.79 L32.46 301.69 L40.39 301.69 ZM32.46 311.79 L21.64 311.79 L21.64 342.55
+ L40.39 342.55 L40.39 311.79 L32.46 311.79 ZM10.82 311.79 L0 311.79 L0 342.55 L18.75 342.55 L18.75
+ 311.79 L10.82 311.79 ZM21.64 311.79 L29.57 311.79 L29.57 301.69 L10.82 301.69 L10.82 311.79
+ L18.75 311.79 L18.75 332.44 L21.64 332.44 L21.64 311.79 Z" class="st10"/>
+ </g>
+ <g id="shape62-276" v:mID="62" v:groupContext="shape" v:layerMember="0" transform="translate(20.0835,-20.5174)">
+ <title>Sheet.62</title>
+ <g id="shadow62-277" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M45.36 341.36 A1.13296 1.18615 -180 1 0 43.09 341.36 A1.13296 1.18615 -180 1 0 45.36 341.36
+ ZM23.46 341.36 A1.13296 1.18615 -180 1 0 21.2 341.36 A1.13296 1.18615 -180 1 0 23.46 341.36
+ ZM2.27 341.36 A1.13296 1.18615 -180 1 0 0 341.36 A1.13296 1.18615 -180 1 0 2.27 341.36 Z"
+ class="st24"/>
+ </g>
+ <path d="M45.36 341.36 A1.13296 1.18615 -180 1 0 43.09 341.36 A1.13296 1.18615 -180 1 0 45.36 341.36 ZM23.46
+ 341.36 A1.13296 1.18615 -180 1 0 21.2 341.36 A1.13296 1.18615 -180 1 0 23.46 341.36 ZM2.27 341.36
+ A1.13296 1.18615 -180 1 0 0 341.36 A1.13296 1.18615 -180 1 0 2.27 341.36 Z" class="st30"/>
+ </g>
+ <g id="shape63-282" v:mID="63" v:groupContext="shape" v:layerMember="0" transform="translate(14.2717,-12.5134)">
+ <title>Sheet.63</title>
+ <v:userDefs>
+ <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(#5b9bd5)"/>
+ </v:userDefs>
+ <g id="shadow63-283" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M43.09 342.55 L51.17 342.55 L51.17 341.74 L43.09 341.74 L43.09 342.55 ZM43.09 340.12 L51.17
+ 340.12 L51.17 339.32 L43.09 339.32 L43.09 340.12 ZM43.09 337.69 L51.17 337.69 L51.17 336.89
+ L43.09 336.89 L43.09 337.69 ZM21.2 342.55 L29.27 342.55 L29.27 341.74 L21.2 341.74 L21.2
+ 342.55 ZM21.2 340.12 L29.27 340.12 L29.27 339.32 L21.2 339.32 L21.2 340.12 ZM21.2 337.69
+ L29.27 337.69 L29.27 336.89 L21.2 336.89 L21.2 337.69 ZM-0 342.55 L8.08 342.55 L8.08 341.74
+ L-0 341.74 L-0 342.55 ZM-0 340.12 L8.08 340.12 L8.08 339.32 L-0 339.32 L-0 340.12 ZM-0 337.69
+ L8.08 337.69 L8.08 336.89 L-0 336.89 L-0 337.69 Z" class="st24"/>
+ </g>
+ <path d="M43.09 342.55 L51.17 342.55 L51.17 341.74 L43.09 341.74 L43.09 342.55 ZM43.09 340.12 L51.17 340.12
+ L51.17 339.32 L43.09 339.32 L43.09 340.12 ZM43.09 337.69 L51.17 337.69 L51.17 336.89 L43.09
+ 336.89 L43.09 337.69 ZM21.2 342.55 L29.27 342.55 L29.27 341.74 L21.2 341.74 L21.2 342.55 ZM21.2
+ 340.12 L29.27 340.12 L29.27 339.32 L21.2 339.32 L21.2 340.12 ZM21.2 337.69 L29.27 337.69 L29.27
+ 336.89 L21.2 336.89 L21.2 337.69 ZM-0 342.55 L8.08 342.55 L8.08 341.74 L-0 341.74 L-0 342.55
+ ZM-0 340.12 L8.08 340.12 L8.08 339.32 L-0 339.32 L-0 340.12 ZM-0 337.69 L8.08 337.69 L8.08 336.89
+ L-0 336.89 L-0 337.69 Z" class="st26"/>
+ </g>
+ </g>
+ <g id="group64-288" transform="translate(59.5234,-47.1858)" v:mID="64" v:groupContext="group">
+ <v:custProps>
+ <v:cp v:nameU="AssetNumber" v:lbl="Asset Number" v:prompt="" v:type="0" v:format="" v:sortKey="Asset"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="SerialNumber" v:lbl="Serial Number" v:prompt="" v:type="0" v:format="" v:sortKey="Asset"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="Location" v:lbl="Location" v:prompt="" v:type="0" v:format="" v:sortKey="Asset"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="Building" v:lbl="Building" v:prompt="" v:type="0" v:format="" v:sortKey="Asset"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="Room" v:lbl="Room" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" v:invis="false"
+ v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="Manufacturer" v:lbl="Manufacturer" v:prompt="" v:type="0" v:format="" v:sortKey="Equipment"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="ProductNumber" v:lbl="Product Number" v:prompt="" v:type="0" v:format=""
+ v:sortKey="Equipment" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="PartNumber" v:lbl="Part Number" v:prompt="" v:type="0" v:format="" v:sortKey="Equipment"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="ProductDescription" v:lbl="Product Description" v:prompt="" v:type="0" v:format=""
+ v:sortKey="Equipment" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="NetworkName" v:lbl="Network Name" v:prompt="" v:type="0" v:format="" v:sortKey="Network"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="IPAddress" v:lbl="IP Address" v:prompt="" v:type="0" v:format="" v:sortKey="Network"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="SubnetMask" v:lbl="Subnet Mask" v:prompt="" v:type="0" v:format="" v:sortKey="Network"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="AdminInterface" v:lbl="Administrative Interface" v:prompt="" v:type="0" v:format=""
+ v:sortKey="Network" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="NumberofPorts" v:lbl="Number of Ports" v:prompt="" v:type="0" v:format=""
+ v:sortKey="Network" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="CommunityString" v:lbl="Community String" v:prompt="" v:type="0" v:format=""
+ v:sortKey="Network" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="NetworkDescription" v:lbl="Network Description" v:prompt="" v:type="0" v:format=""
+ v:sortKey="Network" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="MACAddress" v:lbl="MAC Address" v:prompt="" v:type="0" v:format="" v:sortKey="Network"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="ShapeClass" v:lbl="ShapeClass" v:prompt="" v:type="0" v:format="" v:sortKey=""
+ v:invis="true" v:ask="false" v:langID="1033" v:cal="0" v:val="VT4(Equipment)"/>
+ <v:cp v:nameU="ShapeType" v:lbl="ShapeType" v:prompt="" v:type="0" v:format="" v:sortKey="" v:invis="true"
+ v:ask="false" v:langID="1033" v:cal="0" v:val="VT4(Device)"/>
+ <v:cp v:nameU="SubShapeType" v:lbl="SubShapeType" v:prompt="" v:type="0" v:format="" v:sortKey=""
+ v:invis="true" v:ask="false" v:langID="1033" v:cal="0" v:val="VT4(Load balancer)"/>
+ </v:custProps>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ <v:ud v:nameU="ShapeClass" v:prompt="" v:val="VT0(5):26"/>
+ <v:ud v:nameU="SolSH" v:prompt="" v:val="VT14({BF0433D9-CD73-4EB5-8390-8653BE590246}):41"/>
+ <v:ud v:nameU="visLegendShape" v:prompt="" v:val="VT0(2):26"/>
+ </v:userDefs>
+ <title>Load balancer</title>
+ <g id="shape65-289" v:mID="65" v:groupContext="shape" transform="translate(0,-1.653)">
+ <title>Sheet.65</title>
+ <g id="shadow65-290" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M12.18 329.62 L4.06 329.62 L0 332.02 L0 342.55 L16.23 342.55 L16.23 332.02 L12.18 329.62 Z"
+ class="st24"/>
+ <path d="M0 332.02 L16.23 332.02" class="st25"/>
+ <path d="M12.18 329.62 L4.06 329.62 L0 332.02 L0 342.55 L16.23 342.55 L16.23 332.02 L12.18 329.62"
+ class="st25"/>
+ </g>
+ <path d="M12.18 329.62 L4.06 329.62 L0 332.02 L0 342.55 L16.23 342.55 L16.23 332.02 L12.18 329.62 Z"
+ class="st30"/>
+ <path d="M0 332.02 L16.23 332.02" class="st31"/>
+ <path d="M12.18 329.62 L4.06 329.62 L0 332.02 L0 342.55 L16.23 342.55 L16.23 332.02 L12.18 329.62"
+ class="st31"/>
+ </g>
+ <g id="shape66-297" v:mID="66" v:groupContext="shape" transform="translate(1.81062,-2.91583)">
+ <title>Sheet.66</title>
+ <g id="shadow66-298" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M10.22 341.92 L9.29 342.12 L9.95 342.55 L11.2 342.23 L10.99 340.96 L10.33 340.52 L10.53 341.44
+ L8.34 340.01 L8.03 340.49 L10.22 341.92 ZM11.46 338.22 L8.84 338.22 L8.84 338.78 L11.45
+ 338.78 L10.78 339.45 L11.57 339.45 L12.45 338.5 L11.57 337.55 L10.78 337.55 L11.46 338.22
+ ZM10.48 335.2 L8.29 336.64 L8.6 337.12 L10.79 335.68 L10.59 336.61 L11.25 336.17 L11.46
+ 334.9 L10.21 334.58 L9.55 335.01 L10.48 335.2 ZM6.25 336.37 C5.11 336.37 4.19 337.29 4.19
+ 338.43 C4.19 339.56 5.11 340.48 6.25 340.48 C7.38 340.48 8.31 339.56 8.31 338.43 C8.31 337.29
+ 7.38 336.37 6.25 336.37 ZM6.25 337.02 C7.02 337.02 7.66 337.65 7.66 338.43 C7.66 339.2 7.02
+ 339.83 6.25 339.83 C5.47 339.83 4.84 339.2 4.84 338.43 C4.84 337.65 5.47 337.02 6.25 337.02
+ ZM2.62 338.14 L0 338.14 L0 338.71 L2.62 338.71 L1.94 339.38 L2.74 339.38 L3.61 338.43 L2.73
+ 337.47 L1.95 337.47 L2.62 338.14 Z" class="st9"/>
+ </g>
+ <path d="M10.22 341.92 L9.29 342.12 L9.95 342.55 L11.2 342.23 L10.99 340.96 L10.33 340.52 L10.53 341.44 L8.34
+ 340.01 L8.03 340.49 L10.22 341.92 ZM11.46 338.22 L8.84 338.22 L8.84 338.78 L11.45 338.78 L10.78
+ 339.45 L11.57 339.45 L12.45 338.5 L11.57 337.55 L10.78 337.55 L11.46 338.22 ZM10.48 335.2 L8.29
+ 336.64 L8.6 337.12 L10.79 335.68 L10.59 336.61 L11.25 336.17 L11.46 334.9 L10.21 334.58 L9.55
+ 335.01 L10.48 335.2 ZM6.25 336.37 C5.11 336.37 4.19 337.29 4.19 338.43 C4.19 339.56 5.11 340.48
+ 6.25 340.48 C7.38 340.48 8.31 339.56 8.31 338.43 C8.31 337.29 7.38 336.37 6.25 336.37 ZM6.25
+ 337.02 C7.02 337.02 7.66 337.65 7.66 338.43 C7.66 339.2 7.02 339.83 6.25 339.83 C5.47 339.83
+ 4.84 339.2 4.84 338.43 C4.84 337.65 5.47 337.02 6.25 337.02 ZM2.62 338.14 L0 338.14 L0 338.71
+ L2.62 338.71 L1.94 339.38 L2.74 339.38 L3.61 338.43 L2.73 337.47 L1.95 337.47 L2.62 338.14 Z"
+ class="st32"/>
+ </g>
+ </g>
+ <g id="group67-301" transform="translate(104.617,-86.5795)" v:mID="67" v:groupContext="group">
+ <v:userDefs>
+ <v:ud v:nameU="SkinColor" v:prompt="" v:val="VT5(#da8c36)"/>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <title>Directory server</title>
+ <g id="shape68-302" v:mID="68" v:groupContext="shape" transform="translate(0,-0.451005)">
+ <title>Sheet.68</title>
+ <g id="shadow68-303" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0.47 329.86 L0.47 331.94 L1.46 332.57 L3.33 331.52 L15.43 338.57 L15.43 340.61 L16.42 341.24
+ L18.24 340.22 L22.24 342.55 L22.24 339.27 L36.07 331.28 L36.07 321.27 L19.64 311.85 L3.16
+ 321.13 L3.16 321.5 L0 319.68 L0 329.58 L0.47 329.86 Z" class="st33"/>
+ </g>
+ <path d="M0.47 329.86 L0.47 331.94 L1.46 332.57 L3.33 331.52 L15.43 338.57 L15.43 340.61 L16.42 341.24 L18.24
+ 340.22 L22.24 342.55 L22.24 339.27 L36.07 331.28 L36.07 321.27 L19.64 311.85 L3.16 321.13 L3.16
+ 321.5 L0 319.68 L0 329.58 L0.47 329.86 Z" class="st34"/>
+ </g>
+ <g id="shape69-306" v:mID="69" v:groupContext="shape" transform="translate(3.1636,-11.8063)">
+ <title>Sheet.69</title>
+ <path d="M16.48 323.24 L32.91 332.66 L16.31 342.55 L0 333.26 L0 332.52 L16.48 323.24 Z" class="st35"/>
+ </g>
+ <g id="shape70-310" v:mID="70" v:groupContext="shape" transform="translate(19.06,-3.68954)">
+ <title>Sheet.70</title>
+ <path d="M17.01 324.55 L0 334.19 L3.18 342.55 L17.01 334.56 L17.01 324.55 Z" class="st36"/>
+ </g>
+ <g id="shape71-314" v:mID="71" v:groupContext="shape" transform="translate(0,-0.415652)">
+ <title>Sheet.71</title>
+ <path d="M22.24 342.55 L0 329.58 L0 319.68 L22.24 332.43 L22.24 342.55 Z" class="st37"/>
+ </g>
+ <g id="shape72-322" v:mID="72" v:groupContext="shape" transform="translate(0.82443,-19.8334)">
+ <title>Sheet.72</title>
+ <path d="M1.13 341.58 a0.653986 0.653986 -180 0 0 -0.73971 -0.492434 a0.656072 0.656072 -180 0 0 -0.253101
+ 0.865731 a0.653066 0.653066 -180 0 0 0.740769 0.491375 a0.655459 0.655459 -180 0 0 0.252042
+ -0.864672 Z" class="st38"/>
+ </g>
+ <g id="shape73-326" v:mID="73" v:groupContext="shape" transform="translate(3.62283,-15.1638)">
+ <title>Sheet.73</title>
+ <path d="M3.22 339.78 A1.86495 1.86495 -180 0 0 1.11 338.38 A1.8709 1.8709 -180 0 0 0.38 340.85 A1.86532
+ 1.86532 -180 0 0 2.5 342.25 A1.87264 1.87264 -180 0 0 3.22 339.78 Z" class="st38"/>
+ </g>
+ <g id="shape74-329" v:mID="74" v:groupContext="shape" transform="translate(3.62283,-10.4867)">
+ <title>Sheet.74</title>
+ <path d="M3.22 339.78 A1.86495 1.86495 -180 0 0 1.11 338.38 A1.8709 1.8709 -180 0 0 0.38 340.85 A1.86532
+ 1.86532 -180 0 0 2.5 342.25 A1.87264 1.87264 -180 0 0 3.22 339.78 Z" class="st38"/>
+ </g>
+ <g id="shape75-332" v:mID="75" v:groupContext="shape" transform="translate(4.52404,-16.3668)">
+ <title>Sheet.75</title>
+ <path d="M1.61 341.16 a0.931952 0.931952 -180 0 0 -1.05741 -0.702645 a0.935408 0.935408 -180 0 0 -0.361118
+ 1.23585 a0.932139 0.932139 -180 0 0 1.05794 0.702645 a0.935822 0.935822 -180 0 0 0.360589 -1.23585
+ Z" class="st39"/>
+ </g>
+ <g id="shape76-336" v:mID="76" v:groupContext="shape" transform="translate(4.52404,-11.6897)">
+ <title>Sheet.76</title>
+ <path d="M1.61 341.16 a0.931952 0.931952 -180 0 0 -1.05741 -0.702645 a0.935875 0.935875 -180 0 0 -0.361118
+ 1.23585 a0.932139 0.932139 -180 0 0 1.05794 0.702645 a0.935822 0.935822 -180 0 0 0.360589 -1.23585
+ Z" class="st39"/>
+ </g>
+ <g id="shape77-339" v:mID="77" v:groupContext="shape" transform="translate(7.78787,-8.83469)">
+ <title>Sheet.77</title>
+ <path d="M0 341.57 L0.05 333.6 L1.83 334.57 L1.78 342.55 L0 341.57 Z" class="st40"/>
+ </g>
+ <g id="shape78-343" v:mID="78" v:groupContext="shape" transform="translate(10.204,-7.4008)">
+ <title>Sheet.78</title>
+ <path d="M0 341.57 L0.05 333.6 L1.83 334.57 L1.78 342.55 L0 341.57 Z" class="st40"/>
+ </g>
+ <g id="shape79-346" v:mID="79" v:groupContext="shape" transform="translate(12.6196,-5.96639)">
+ <title>Sheet.79</title>
+ <path d="M0 341.57 L0.05 333.6 L1.83 334.57 L1.78 342.55 L0 341.57 Z" class="st40"/>
+ </g>
+ <g id="shape80-349" v:mID="80" v:groupContext="shape" transform="translate(15.0357,-4.53251)">
+ <title>Sheet.80</title>
+ <path d="M0 341.57 L0.05 333.6 L1.83 334.57 L1.78 342.55 L0 341.57 Z" class="st40"/>
+ </g>
+ <g id="shape81-352" v:mID="81" v:groupContext="shape" transform="translate(8.24006,-10.0631)">
+ <title>Sheet.81</title>
+ <path d="M0.85 342.24 a0.388199 0.388199 0 0 1 -0.425188 0.308698 a0.638045 0.638045 0 0 1 -0.424658 -0.573447
+ L0 336.5 a0.387575 0.387575 0 0 1 0.424658 -0.308698 a0.637725 0.637725 0 0 1 0.425188 0.573447
+ L0.85 342.24 Z" class="st41"/>
+ </g>
+ <g id="shape82-356" v:mID="82" v:groupContext="shape" transform="translate(10.6556,-8.62924)">
+ <title>Sheet.82</title>
+ <path d="M0.85 342.24 a0.388199 0.388199 0 0 1 -0.425188 0.308698 a0.638045 0.638045 0 0 1 -0.424658 -0.573447
+ L0 336.5 a0.387575 0.387575 0 0 1 0.424658 -0.308698 a0.637725 0.637725 0 0 1 0.425188 0.573447
+ L0.85 342.24 Z" class="st41"/>
+ </g>
+ <g id="shape83-359" v:mID="83" v:groupContext="shape" transform="translate(13.0717,-7.19483)">
+ <title>Sheet.83</title>
+ <path d="M0.85 342.24 a0.388199 0.388199 0 0 1 -0.425188 0.308698 a0.638045 0.638045 0 0 1 -0.424658 -0.573447
+ L0 336.5 a0.387575 0.387575 0 0 1 0.424658 -0.308698 a0.637725 0.637725 0 0 1 0.425188 0.573447
+ L0.85 342.24 Z" class="st41"/>
+ </g>
+ <g id="shape84-362" v:mID="84" v:groupContext="shape" transform="translate(15.4873,-5.76095)">
+ <title>Sheet.84</title>
+ <path d="M0.85 342.24 a0.388502 0.388502 0 0 1 -0.425717 0.308698 a0.638367 0.638367 0 0 1 -0.424129 -0.573447
+ L0 336.5 a0.387272 0.387272 0 0 1 0.424129 -0.308698 a0.638235 0.638235 0 0 1 0.425717 0.573447
+ L0.85 342.24 Z" class="st41"/>
+ </g>
+ <g id="shape85-365" v:mID="85" v:groupContext="shape" transform="translate(7.78787,-9.81214)">
+ <title>Sheet.85</title>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55 L0 342.55 Z" class="st42"/>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55" class="st43"/>
+ </g>
+ <g id="shape86-368" v:mID="86" v:groupContext="shape" transform="translate(10.204,-8.37826)">
+ <title>Sheet.86</title>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55 L0 342.55 Z" class="st42"/>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55" class="st43"/>
+ </g>
+ <g id="shape87-371" v:mID="87" v:groupContext="shape" transform="translate(12.6196,-6.94385)">
+ <title>Sheet.87</title>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55 L0 342.55 Z" class="st42"/>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55" class="st43"/>
+ </g>
+ <g id="shape88-374" v:mID="88" v:groupContext="shape" transform="translate(15.0357,-5.50996)">
+ <title>Sheet.88</title>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55 L0 342.55 Z" class="st42"/>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55" class="st43"/>
+ </g>
+ <g id="shape89-377" v:mID="89" v:groupContext="shape" transform="translate(7.78787,-4.53251)">
+ <title>Sheet.89</title>
+ <path d="M9.08 334.57 L9.03 342.55 L7.25 341.57 L9.08 334.57 ZM6.66 333.14 L6.61 341.11 L4.83 340.13 L6.66
+ 333.14 ZM4.25 331.7 L4.2 339.68 L2.42 338.7 L4.25 331.7 ZM1.83 330.27 L1.78 338.24 L0 337.27
+ L1.83 330.27 Z" class="st42"/>
+ <path d="M9.08 334.57 L9.03 342.55 L7.25 341.57M6.66 333.14 L6.61 341.11 L4.83 340.13M4.25 331.7 L4.2 339.68
+ L2.42 338.7M1.83 330.27 L1.78 338.24 L0 337.27" class="st44"/>
+ </g>
+ <g id="shape90-380" v:mID="90" v:groupContext="shape" transform="translate(2.22125,-11.8454)">
+ <title>Sheet.90</title>
+ <path d="M0 341.85 L0.63 341.42 L1.42 341.78 L0.03 342.55 L0 341.85 Z" class="st45"/>
+ </g>
+ <g id="shape91-384" v:mID="91" v:groupContext="shape" transform="translate(17.1796,-3.17487)">
+ <title>Sheet.91</title>
+ <path d="M0 341.85 L0.63 341.42 L1.42 341.78 L0.03 342.55 L0 341.85 Z" class="st45"/>
+ </g>
+ <g id="shape92-387" v:mID="92" v:groupContext="shape" transform="translate(1.46036,-10.3893)">
+ <title>Sheet.92</title>
+ <path d="M2.12 341.35 L0 342.55 L0 333.54 L2.12 332.29 L2.12 333.41 L0.79 334.15 L0.79 341.09 L2.18 340.33
+ L2.12 341.35 Z" class="st36"/>
+ </g>
+ <g id="shape93-390" v:mID="93" v:groupContext="shape" transform="translate(16.4187,-1.71875)">
+ <title>Sheet.93</title>
+ <path d="M2.12 341.35 L0 342.55 L0 333.54 L2.12 332.29 L2.12 333.41 L0.79 334.15 L0.79 341.09 L2.18 340.33
+ L2.12 341.35 Z" class="st36"/>
+ </g>
+ <g id="shape94-393" v:mID="94" v:groupContext="shape" transform="translate(0.467548,-10.3893)">
+ <title>Sheet.94</title>
+ <path d="M0.99 333.54 L3.11 332.29 L2.12 331.66 L0 332.91 L0 341.92 L0.99 342.55 L0.99 333.54 Z"
+ class="st46"/>
+ </g>
+ <g id="shape95-397" v:mID="95" v:groupContext="shape" transform="translate(15.4259,-1.71875)">
+ <title>Sheet.95</title>
+ <path d="M0.99 333.54 L3.11 332.29 L2.12 331.66 L0 332.91 L0 341.92 L0.99 342.55 L0.99 333.54 Z"
+ class="st46"/>
+ </g>
+ <g id="shape96-400" v:mID="96" v:groupContext="shape" transform="translate(0.467548,-1.71928)">
+ <title>Sheet.96</title>
+ <path d="M17.34 339.96 L16.75 340.37 L16.75 334.15 L18.07 333.41 L18.07 332.29 L17.08 331.66 L14.96 332.91
+ L14.96 341.92 L15.95 342.55 L18.07 341.35 L18.14 340.33 L17.34 339.96 ZM2.38 331.29 L1.79 331.7
+ L1.79 325.48 L3.11 324.74 L3.11 323.62 L2.12 322.99 L0 324.24 L0 333.25 L0.99 333.87 L3.11 332.68
+ L3.18 331.66 L2.38 331.29 Z" class="st47"/>
+ </g>
+ <g id="shape97-402" v:mID="97" v:groupContext="shape" transform="translate(19.9526,-8.71396)">
+ <title>Sheet.97</title>
+ <path d="M1.13 341.58 a0.653986 0.653986 -180 0 0 -0.73971 -0.492434 a0.656072 0.656072 -180 0 0 -0.253101
+ 0.865731 a0.653066 0.653066 -180 0 0 0.740769 0.491375 a0.655459 0.655459 -180 0 0 0.252042
+ -0.864672 Z" class="st38"/>
+ </g>
+ <g id="shape98-405" v:mID="98" v:groupContext="shape" transform="translate(19.9526,-2.35997)">
+ <title>Sheet.98</title>
+ <path d="M1.13 341.58 a0.653986 0.653986 -180 0 0 -0.73971 -0.492434 a0.656072 0.656072 -180 0 0 -0.253101
+ 0.865731 a0.653066 0.653066 -180 0 0 0.740769 0.491375 a0.655459 0.655459 -180 0 0 0.252042
+ -0.864672 Z" class="st38"/>
+ </g>
+ <g id="shape99-408" v:mID="99" v:groupContext="shape" transform="translate(0,-0.415652)">
+ <title>Sheet.99</title>
+ <path d="M36.07 331.28 L36.07 321.27 L19.64 311.85 L3.16 321.13 L3.16 321.52 L0 319.68 L0 329.58 L0.47 329.86
+ L0.47 331.94 L1.46 332.57 L3.33 331.52 L15.43 338.57 L15.43 340.61 L16.42 341.24 L18.24 340.22
+ L22.24 342.55 L22.24 339.27 L36.07 331.28 Z" class="st47"/>
+ </g>
+ <g id="shape100-410" v:mID="100" v:groupContext="shape" transform="translate(27.8077,-2.86477)">
+ <title>Sheet.100</title>
+ <path d="M0.29 342.55 L6.62 338.89 A1.82805 1.82805 0 0 1 6.62 336.9 L0.29 340.55 A1.82805 1.82805 -180 0
+ 0 0.29 342.55 Z" class="st48"/>
+ </g>
+ <g id="shape101-412" v:mID="101" v:groupContext="shape" transform="translate(23.5035,-4.85627)">
+ <title>Sheet.101</title>
+ <path d="M4.6 342.55 L10.92 338.89 L6.32 336.24 L0 339.89 L4.6 342.55 Z" class="st49"/>
+ </g>
+ <g id="shape102-416" v:mID="102" v:groupContext="shape" transform="translate(23.3588,-2.86477)">
+ <title>Sheet.102</title>
+ <path d="M0.14 339.89 L4.74 342.55 A1.82805 1.82805 0 0 1 4.74 340.55 L0.14 337.9 A3.49826 3.49826 -180 0
+ 0 0.14 339.89 Z" class="st50"/>
+ </g>
+ <g id="shape103-420" v:mID="103" v:groupContext="shape" transform="translate(25.8933,-5.98478)">
+ <title>Sheet.103</title>
+ <path d="M2.87 342.55 L0 340.89" class="st51"/>
+ <path d="M0.94 340.34 L3.82 342" class="st51"/>
+ <path d="M1.88 339.8 L4.76 341.46" class="st51"/>
+ <path d="M2.82 339.26 L5.7 340.92" class="st51"/>
+ <path d="M3.76 338.71 L6.64 340.37" class="st51"/>
+ </g>
+ <g id="shape104-427" v:mID="104" v:groupContext="shape" transform="translate(23.5035,-7.51159)">
+ <title>Sheet.104</title>
+ <path d="M5.13 341.17 L11.45 337.52 A11.9345 11.9345 0 0 1 6.32 338.89 L0 342.55 A11.9345 11.9345 -180 0
+ 0 5.13 341.17 Z" class="st52"/>
+ </g>
+ <g id="shape105-435" v:mID="105" v:groupContext="shape" transform="translate(30.2106,-4.74563)">
+ <title>Sheet.105</title>
+ <path d="M0.98 341.98 L0 342.55" class="st51"/>
+ <path d="M1.26 341.48 L2.24 340.92" class="st51"/>
+ <path d="M2.53 340.42 L3.51 339.86" class="st51"/>
+ </g>
+ <g id="shape106-440" v:mID="106" v:groupContext="shape" transform="translate(23.3588,-2.86477)">
+ <title>Sheet.106</title>
+ <path d="M0.14 339.89 L4.74 342.55 L11.07 338.89 A1.82805 1.82805 0 0 1 11.07 336.9 L7.85 335.04 L11.6 332.87
+ A11.9345 11.9345 0 0 1 6.47 334.25 L0.14 337.9 A3.49826 3.49826 -180 0 0 0.14 339.89"
+ class="st53"/>
+ </g>
+ </g>
+ <g id="group107-443" transform="translate(104.617,-33.8201)" v:mID="107" v:groupContext="group">
+ <v:userDefs>
+ <v:ud v:nameU="SkinColor" v:prompt="" v:val="VT5(#da8c36)"/>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <title>Directory server.104</title>
+ <g id="shape108-444" v:mID="108" v:groupContext="shape" transform="translate(0,-0.451005)">
+ <title>Sheet.108</title>
+ <g id="shadow108-445" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0.47 329.86 L0.47 331.94 L1.46 332.57 L3.33 331.52 L15.43 338.57 L15.43 340.61 L16.42 341.24
+ L18.24 340.22 L22.24 342.55 L22.24 339.27 L36.07 331.28 L36.07 321.27 L19.64 311.85 L3.16
+ 321.13 L3.16 321.5 L0 319.68 L0 329.58 L0.47 329.86 Z" class="st33"/>
+ </g>
+ <path d="M0.47 329.86 L0.47 331.94 L1.46 332.57 L3.33 331.52 L15.43 338.57 L15.43 340.61 L16.42 341.24 L18.24
+ 340.22 L22.24 342.55 L22.24 339.27 L36.07 331.28 L36.07 321.27 L19.64 311.85 L3.16 321.13 L3.16
+ 321.5 L0 319.68 L0 329.58 L0.47 329.86 Z" class="st34"/>
+ </g>
+ <g id="shape109-448" v:mID="109" v:groupContext="shape" transform="translate(3.1636,-11.8063)">
+ <title>Sheet.109</title>
+ <path d="M16.48 323.24 L32.91 332.66 L16.31 342.55 L0 333.26 L0 332.52 L16.48 323.24 Z" class="st35"/>
+ </g>
+ <g id="shape110-451" v:mID="110" v:groupContext="shape" transform="translate(19.06,-3.68954)">
+ <title>Sheet.110</title>
+ <path d="M17.01 324.55 L0 334.19 L3.18 342.55 L17.01 334.56 L17.01 324.55 Z" class="st36"/>
+ </g>
+ <g id="shape111-454" v:mID="111" v:groupContext="shape" transform="translate(0,-0.415652)">
+ <title>Sheet.111</title>
+ <path d="M22.24 342.55 L0 329.58 L0 319.68 L22.24 332.43 L22.24 342.55 Z" class="st37"/>
+ </g>
+ <g id="shape112-457" v:mID="112" v:groupContext="shape" transform="translate(0.82443,-19.8334)">
+ <title>Sheet.112</title>
+ <path d="M1.13 341.58 a0.653986 0.653986 -180 0 0 -0.73971 -0.492434 a0.656072 0.656072 -180 0 0 -0.253101
+ 0.865731 a0.653066 0.653066 -180 0 0 0.740769 0.491375 a0.655459 0.655459 -180 0 0 0.252042
+ -0.864672 Z" class="st38"/>
+ </g>
+ <g id="shape113-460" v:mID="113" v:groupContext="shape" transform="translate(3.62283,-15.1638)">
+ <title>Sheet.113</title>
+ <path d="M3.22 339.78 A1.86495 1.86495 -180 0 0 1.11 338.38 A1.8709 1.8709 -180 0 0 0.38 340.85 A1.86532
+ 1.86532 -180 0 0 2.5 342.25 A1.87264 1.87264 -180 0 0 3.22 339.78 Z" class="st38"/>
+ </g>
+ <g id="shape114-463" v:mID="114" v:groupContext="shape" transform="translate(3.62283,-10.4867)">
+ <title>Sheet.114</title>
+ <path d="M3.22 339.78 A1.86495 1.86495 -180 0 0 1.11 338.38 A1.8709 1.8709 -180 0 0 0.38 340.85 A1.86532
+ 1.86532 -180 0 0 2.5 342.25 A1.87264 1.87264 -180 0 0 3.22 339.78 Z" class="st38"/>
+ </g>
+ <g id="shape115-466" v:mID="115" v:groupContext="shape" transform="translate(4.52404,-16.3668)">
+ <title>Sheet.115</title>
+ <path d="M1.61 341.16 a0.931952 0.931952 -180 0 0 -1.05741 -0.702645 a0.935408 0.935408 -180 0 0 -0.361118
+ 1.23585 a0.932139 0.932139 -180 0 0 1.05794 0.702645 a0.935822 0.935822 -180 0 0 0.360589 -1.23585
+ Z" class="st39"/>
+ </g>
+ <g id="shape116-469" v:mID="116" v:groupContext="shape" transform="translate(4.52404,-11.6897)">
+ <title>Sheet.116</title>
+ <path d="M1.61 341.16 a0.931952 0.931952 -180 0 0 -1.05741 -0.702645 a0.935875 0.935875 -180 0 0 -0.361118
+ 1.23585 a0.932139 0.932139 -180 0 0 1.05794 0.702645 a0.935822 0.935822 -180 0 0 0.360589 -1.23585
+ Z" class="st39"/>
+ </g>
+ <g id="shape117-472" v:mID="117" v:groupContext="shape" transform="translate(7.78787,-8.83469)">
+ <title>Sheet.117</title>
+ <path d="M0 341.57 L0.05 333.6 L1.83 334.57 L1.78 342.55 L0 341.57 Z" class="st40"/>
+ </g>
+ <g id="shape118-475" v:mID="118" v:groupContext="shape" transform="translate(10.204,-7.4008)">
+ <title>Sheet.118</title>
+ <path d="M0 341.57 L0.05 333.6 L1.83 334.57 L1.78 342.55 L0 341.57 Z" class="st40"/>
+ </g>
+ <g id="shape119-478" v:mID="119" v:groupContext="shape" transform="translate(12.6196,-5.96639)">
+ <title>Sheet.119</title>
+ <path d="M0 341.57 L0.05 333.6 L1.83 334.57 L1.78 342.55 L0 341.57 Z" class="st40"/>
+ </g>
+ <g id="shape120-481" v:mID="120" v:groupContext="shape" transform="translate(15.0357,-4.53251)">
+ <title>Sheet.120</title>
+ <path d="M0 341.57 L0.05 333.6 L1.83 334.57 L1.78 342.55 L0 341.57 Z" class="st40"/>
+ </g>
+ <g id="shape121-484" v:mID="121" v:groupContext="shape" transform="translate(8.24006,-10.0631)">
+ <title>Sheet.121</title>
+ <path d="M0.85 342.24 a0.388199 0.388199 0 0 1 -0.425188 0.308698 a0.638045 0.638045 0 0 1 -0.424658 -0.573447
+ L0 336.5 a0.387575 0.387575 0 0 1 0.424658 -0.308698 a0.637725 0.637725 0 0 1 0.425188 0.573447
+ L0.85 342.24 Z" class="st41"/>
+ </g>
+ <g id="shape122-487" v:mID="122" v:groupContext="shape" transform="translate(10.6556,-8.62924)">
+ <title>Sheet.122</title>
+ <path d="M0.85 342.24 a0.388199 0.388199 0 0 1 -0.425188 0.308698 a0.638045 0.638045 0 0 1 -0.424658 -0.573447
+ L0 336.5 a0.387575 0.387575 0 0 1 0.424658 -0.308698 a0.637725 0.637725 0 0 1 0.425188 0.573447
+ L0.85 342.24 Z" class="st41"/>
+ </g>
+ <g id="shape123-490" v:mID="123" v:groupContext="shape" transform="translate(13.0717,-7.19483)">
+ <title>Sheet.123</title>
+ <path d="M0.85 342.24 a0.388199 0.388199 0 0 1 -0.425188 0.308698 a0.638045 0.638045 0 0 1 -0.424658 -0.573447
+ L0 336.5 a0.387575 0.387575 0 0 1 0.424658 -0.308698 a0.637725 0.637725 0 0 1 0.425188 0.573447
+ L0.85 342.24 Z" class="st41"/>
+ </g>
+ <g id="shape124-493" v:mID="124" v:groupContext="shape" transform="translate(15.4873,-5.76095)">
+ <title>Sheet.124</title>
+ <path d="M0.85 342.24 a0.388502 0.388502 0 0 1 -0.425717 0.308698 a0.638367 0.638367 0 0 1 -0.424129 -0.573447
+ L0 336.5 a0.387272 0.387272 0 0 1 0.424129 -0.308698 a0.638235 0.638235 0 0 1 0.425717 0.573447
+ L0.85 342.24 Z" class="st41"/>
+ </g>
+ <g id="shape125-496" v:mID="125" v:groupContext="shape" transform="translate(7.78787,-9.81214)">
+ <title>Sheet.125</title>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55 L0 342.55 Z" class="st42"/>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55" class="st43"/>
+ </g>
+ <g id="shape126-499" v:mID="126" v:groupContext="shape" transform="translate(10.204,-8.37826)">
+ <title>Sheet.126</title>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55 L0 342.55 Z" class="st42"/>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55" class="st43"/>
+ </g>
+ <g id="shape127-502" v:mID="127" v:groupContext="shape" transform="translate(12.6196,-6.94385)">
+ <title>Sheet.127</title>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55 L0 342.55 Z" class="st42"/>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55" class="st43"/>
+ </g>
+ <g id="shape128-505" v:mID="128" v:groupContext="shape" transform="translate(15.0357,-5.50996)">
+ <title>Sheet.128</title>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55 L0 342.55 Z" class="st42"/>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55" class="st43"/>
+ </g>
+ <g id="shape129-508" v:mID="129" v:groupContext="shape" transform="translate(7.78787,-4.53251)">
+ <title>Sheet.129</title>
+ <path d="M9.08 334.57 L9.03 342.55 L7.25 341.57 L9.08 334.57 ZM6.66 333.14 L6.61 341.11 L4.83 340.13 L6.66
+ 333.14 ZM4.25 331.7 L4.2 339.68 L2.42 338.7 L4.25 331.7 ZM1.83 330.27 L1.78 338.24 L0 337.27
+ L1.83 330.27 Z" class="st42"/>
+ <path d="M9.08 334.57 L9.03 342.55 L7.25 341.57M6.66 333.14 L6.61 341.11 L4.83 340.13M4.25 331.7 L4.2 339.68
+ L2.42 338.7M1.83 330.27 L1.78 338.24 L0 337.27" class="st44"/>
+ </g>
+ <g id="shape130-511" v:mID="130" v:groupContext="shape" transform="translate(2.22125,-11.8454)">
+ <title>Sheet.130</title>
+ <path d="M0 341.85 L0.63 341.42 L1.42 341.78 L0.03 342.55 L0 341.85 Z" class="st45"/>
+ </g>
+ <g id="shape131-514" v:mID="131" v:groupContext="shape" transform="translate(17.1796,-3.17487)">
+ <title>Sheet.131</title>
+ <path d="M0 341.85 L0.63 341.42 L1.42 341.78 L0.03 342.55 L0 341.85 Z" class="st45"/>
+ </g>
+ <g id="shape132-517" v:mID="132" v:groupContext="shape" transform="translate(1.46036,-10.3893)">
+ <title>Sheet.132</title>
+ <path d="M2.12 341.35 L0 342.55 L0 333.54 L2.12 332.29 L2.12 333.41 L0.79 334.15 L0.79 341.09 L2.18 340.33
+ L2.12 341.35 Z" class="st36"/>
+ </g>
+ <g id="shape133-520" v:mID="133" v:groupContext="shape" transform="translate(16.4187,-1.71875)">
+ <title>Sheet.133</title>
+ <path d="M2.12 341.35 L0 342.55 L0 333.54 L2.12 332.29 L2.12 333.41 L0.79 334.15 L0.79 341.09 L2.18 340.33
+ L2.12 341.35 Z" class="st36"/>
+ </g>
+ <g id="shape134-523" v:mID="134" v:groupContext="shape" transform="translate(0.467548,-10.3893)">
+ <title>Sheet.134</title>
+ <path d="M0.99 333.54 L3.11 332.29 L2.12 331.66 L0 332.91 L0 341.92 L0.99 342.55 L0.99 333.54 Z"
+ class="st46"/>
+ </g>
+ <g id="shape135-526" v:mID="135" v:groupContext="shape" transform="translate(15.4259,-1.71875)">
+ <title>Sheet.135</title>
+ <path d="M0.99 333.54 L3.11 332.29 L2.12 331.66 L0 332.91 L0 341.92 L0.99 342.55 L0.99 333.54 Z"
+ class="st46"/>
+ </g>
+ <g id="shape136-529" v:mID="136" v:groupContext="shape" transform="translate(0.467548,-1.71928)">
+ <title>Sheet.136</title>
+ <path d="M17.34 339.96 L16.75 340.37 L16.75 334.15 L18.07 333.41 L18.07 332.29 L17.08 331.66 L14.96 332.91
+ L14.96 341.92 L15.95 342.55 L18.07 341.35 L18.14 340.33 L17.34 339.96 ZM2.38 331.29 L1.79 331.7
+ L1.79 325.48 L3.11 324.74 L3.11 323.62 L2.12 322.99 L0 324.24 L0 333.25 L0.99 333.87 L3.11 332.68
+ L3.18 331.66 L2.38 331.29 Z" class="st47"/>
+ </g>
+ <g id="shape137-531" v:mID="137" v:groupContext="shape" transform="translate(19.9526,-8.71396)">
+ <title>Sheet.137</title>
+ <path d="M1.13 341.58 a0.653986 0.653986 -180 0 0 -0.73971 -0.492434 a0.656072 0.656072 -180 0 0 -0.253101
+ 0.865731 a0.653066 0.653066 -180 0 0 0.740769 0.491375 a0.655459 0.655459 -180 0 0 0.252042
+ -0.864672 Z" class="st38"/>
+ </g>
+ <g id="shape138-534" v:mID="138" v:groupContext="shape" transform="translate(19.9526,-2.35997)">
+ <title>Sheet.138</title>
+ <path d="M1.13 341.58 a0.653986 0.653986 -180 0 0 -0.73971 -0.492434 a0.656072 0.656072 -180 0 0 -0.253101
+ 0.865731 a0.653066 0.653066 -180 0 0 0.740769 0.491375 a0.655459 0.655459 -180 0 0 0.252042
+ -0.864672 Z" class="st38"/>
+ </g>
+ <g id="shape139-537" v:mID="139" v:groupContext="shape" transform="translate(0,-0.415652)">
+ <title>Sheet.139</title>
+ <path d="M36.07 331.28 L36.07 321.27 L19.64 311.85 L3.16 321.13 L3.16 321.52 L0 319.68 L0 329.58 L0.47 329.86
+ L0.47 331.94 L1.46 332.57 L3.33 331.52 L15.43 338.57 L15.43 340.61 L16.42 341.24 L18.24 340.22
+ L22.24 342.55 L22.24 339.27 L36.07 331.28 Z" class="st47"/>
+ </g>
+ <g id="shape140-539" v:mID="140" v:groupContext="shape" transform="translate(27.8077,-2.86477)">
+ <title>Sheet.140</title>
+ <path d="M0.29 342.55 L6.62 338.89 A1.82805 1.82805 0 0 1 6.62 336.9 L0.29 340.55 A1.82805 1.82805 -180 0
+ 0 0.29 342.55 Z" class="st48"/>
+ </g>
+ <g id="shape141-541" v:mID="141" v:groupContext="shape" transform="translate(23.5035,-4.85627)">
+ <title>Sheet.141</title>
+ <path d="M4.6 342.55 L10.92 338.89 L6.32 336.24 L0 339.89 L4.6 342.55 Z" class="st49"/>
+ </g>
+ <g id="shape142-544" v:mID="142" v:groupContext="shape" transform="translate(23.3588,-2.86477)">
+ <title>Sheet.142</title>
+ <path d="M0.14 339.89 L4.74 342.55 A1.82805 1.82805 0 0 1 4.74 340.55 L0.14 337.9 A3.49826 3.49826 -180 0
+ 0 0.14 339.89 Z" class="st50"/>
+ </g>
+ <g id="shape143-547" v:mID="143" v:groupContext="shape" transform="translate(25.8933,-5.98478)">
+ <title>Sheet.143</title>
+ <path d="M2.87 342.55 L0 340.89" class="st51"/>
+ <path d="M0.94 340.34 L3.82 342" class="st51"/>
+ <path d="M1.88 339.8 L4.76 341.46" class="st51"/>
+ <path d="M2.82 339.26 L5.7 340.92" class="st51"/>
+ <path d="M3.76 338.71 L6.64 340.37" class="st51"/>
+ </g>
+ <g id="shape144-554" v:mID="144" v:groupContext="shape" transform="translate(23.5035,-7.51159)">
+ <title>Sheet.144</title>
+ <path d="M5.13 341.17 L11.45 337.52 A11.9345 11.9345 0 0 1 6.32 338.89 L0 342.55 A11.9345 11.9345 -180 0
+ 0 5.13 341.17 Z" class="st52"/>
+ </g>
+ <g id="shape145-557" v:mID="145" v:groupContext="shape" transform="translate(30.2106,-4.74563)">
+ <title>Sheet.145</title>
+ <path d="M0.98 341.98 L0 342.55" class="st51"/>
+ <path d="M1.26 341.48 L2.24 340.92" class="st51"/>
+ <path d="M2.53 340.42 L3.51 339.86" class="st51"/>
+ </g>
+ <g id="shape146-562" v:mID="146" v:groupContext="shape" transform="translate(23.3588,-2.86477)">
+ <title>Sheet.146</title>
+ <path d="M0.14 339.89 L4.74 342.55 L11.07 338.89 A1.82805 1.82805 0 0 1 11.07 336.9 L7.85 335.04 L11.6 332.87
+ A11.9345 11.9345 0 0 1 6.47 334.25 L0.14 337.9 A3.49826 3.49826 -180 0 0 0.14 339.89"
+ class="st53"/>
+ </g>
+ </g>
+ <g id="shape147-565" v:mID="147" v:groupContext="shape" transform="translate(427.321,214.49) rotate(90)">
+ <title>Cloud</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <path d="M5.37 311.54 A8.61618 10.0654 0 0 1 9.5 292.2 A17.4727 20.4114 0 0 1 34.86 275.89 A20.0634 23.4379 0
+ 0 1 56.58 272.26 A12.5816 14.6977 0 0 1 75.21 271.05 A14.3244 16.7336 0 0 1 97.98 277.09 A10.2423
+ 11.9646 0 0 1 106.25 294.02 A12.6864 14.8197 0 0 1 95.9 318.19 A16.0049 18.6962 0 0 1 73.14 330.27
+ A18.8712 22.0444 0 0 1 42.1 335.11 A23.9217 27.9441 0 0 1 15.2 330.27 A9.43759 11.0249 0 0 1 5.37
+ 311.54 Z" class="st42"/>
+ <path d="M5.37 311.54 A8.61618 10.0654 0 0 1 9.5 292.2 A17.4727 20.4114 0 0 1 34.86 275.89 A20.0634 23.4379 0
+ 0 1 56.58 272.26 A12.5816 14.6977 0 0 1 75.21 271.05 A14.3244 16.7336 0 0 1 97.98 277.09 A10.2423
+ 11.9646 0 0 1 106.25 294.02 A12.6864 14.8197 0 0 1 95.9 318.19 A16.0049 18.6962 0 0 1 73.14 330.27
+ A18.8712 22.0444 0 0 1 42.1 335.11 A23.9217 27.9441 0 0 1 15.2 330.27 A9.43759 11.0249 0 0 1 5.37
+ 311.54" class="st54"/>
+ <path d="M11.05 312.14 A8.59237 10.0375 0 0 1 5.37 311.54" class="st54"/>
+ <path d="M40.54 332.09 A8.62978 10.0812 -180 0 0 42.1 335.11" class="st54"/>
+ <path d="M73.92 326.65 A6.96633 8.13801 0 0 1 73.14 330.27" class="st54"/>
+ <path d="M89.7 308.52 A7.30994 8.5394 0 0 1 95.9 318.19" class="st54"/>
+ <path d="M103.15 297.64 A6.67364 7.79609 -180 0 0 106.25 294.02" class="st54"/>
+ <path d="M37.96 278.3 A10.2914 12.0219 -180 0 0 34.86 275.89" class="st54"/>
+ </g>
+ <g id="shape148-574" v:mID="148" v:groupContext="shape" transform="translate(110.222,-64.9346)">
+ <title>Triangle</title>
+ <desc>setsum</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="15.6995" cy="336.449" width="31.4" height="12.1933"/>
+ <g id="shadow148-575" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M31.4 342.55 L14.67 324.26 L0 342.55 L31.4 342.55 Z" class="st9"/>
+ </g>
+ <path d="M31.4 342.55 L14.67 324.26 L0 342.55 L31.4 342.55 Z" class="st10"/>
+ <text x="8.35" y="337.95" class="st55" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>setsum</text> </g>
+ <g id="shape149-579" v:mID="149" v:groupContext="shape" transform="translate(292.639,20.8827) rotate(45)">
+ <title>Sheet.149</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L14.71 342.55" class="st14"/>
+ </g>
+ <g id="shape150-584" v:mID="150" v:groupContext="shape" transform="translate(43.2897,-54.1122)">
+ <title>Sheet.150</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L10.07 342.55" class="st14"/>
+ </g>
+ <g id="shape151-589" v:mID="151" v:groupContext="shape" transform="translate(-112.261,8.34531) rotate(-28.1394)">
+ <title>Sheet.151</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L18 342.55" class="st14"/>
+ </g>
+ <g id="shape152-594" v:mID="152" v:groupContext="shape">
+ <title>Sheet.152</title>
+ <desc>Clients</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="32.4673" cy="337.134" width="64.94" height="10.8224"/>
+ <rect x="0" y="331.723" width="64.9346" height="10.8224" class="st11"/>
+ <text x="21.5" y="339.53" class="st6" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Clients</text> </g>
+ <g id="shape153-597" v:mID="153" v:groupContext="shape" transform="translate(83.578,-9.58078)">
+ <title>Sheet.153</title>
+ <desc>Distributed Cache</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="42.0677" cy="337.134" width="84.14" height="10.8224"/>
+ <rect x="0" y="331.723" width="84.1355" height="10.8224" class="st11"/>
+ <text x="13.1" y="339.53" class="st6" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Distributed Cache</text> </g>
+ <g id="shape154-600" v:mID="154" v:groupContext="shape" transform="translate(181.983,-18.6146)">
+ <title>Sheet.154</title>
+ <desc>Web Servers</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="32.4673" cy="337.134" width="64.94" height="10.8224"/>
+ <rect x="0" y="331.723" width="64.9346" height="10.8224" class="st11"/>
+ <text x="11.93" y="339.53" class="st6" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Web Servers</text> </g>
+ <g id="shape155-603" v:mID="155" v:groupContext="shape" transform="translate(96.6068,630.978) rotate(180)">
+ <title>Simple Arrow</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="ArrowType" v:prompt="" v:val="VT0(2):26"/>
+ </v:userDefs>
+ <g id="shadow155-604" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,-0.3456,-1.9728)" class="st1">
+ <path d="M0 342.55 L12 330.55 L12 336.55 L16.23 336.55 L16.23 342.55 L16.23 348.54 L12 348.54 L12 354.54
+ L0 342.55 Z" class="st56"/>
+ </g>
+ <path d="M0 342.55 L12 330.55 L12 336.55 L16.23 336.55 L16.23 342.55 L16.23 348.54 L12 348.54 L12 354.54 L0 342.55
+ Z" class="st57"/>
+ </g>
+ <g id="shape156-607" v:mID="156" v:groupContext="shape" transform="translate(173.159,625.567) rotate(180)">
+ <title>Simple Arrow.153</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="ArrowType" v:prompt="" v:val="VT0(2):26"/>
+ </v:userDefs>
+ <g id="shadow156-608" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,-0.3456,-1.9728)" class="st1">
+ <path d="M0 342.55 L12 330.55 L12 336.55 L16.23 336.55 L16.23 342.55 L16.23 348.54 L12 348.54 L12 354.54
+ L0 342.55 Z" class="st56"/>
+ </g>
+ <path d="M0 342.55 L12 330.55 L12 336.55 L16.23 336.55 L16.23 342.55 L16.23 348.54 L12 348.54 L12 354.54 L0 342.55
+ Z" class="st57"/>
+ </g>
+ </g>
+ <g id="shape157-611" v:mID="157" v:groupContext="shape" transform="translate(0,-149.475)">
+ <title>Rectangle</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow157-612" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="193.823" width="271.116" height="148.722" class="st58"/>
+ </g>
+ <rect x="0" y="193.823" width="271.116" height="148.722" class="st59"/>
+ </g>
+ <g id="shape158-615" v:mID="158" v:groupContext="shape" transform="translate(271.116,-149.475)">
+ <title>Rectangle.158</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow158-616" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="193.823" width="202.104" height="148.722" class="st58"/>
+ </g>
+ <rect x="0" y="193.823" width="202.104" height="148.722" class="st59"/>
+ </g>
+ <g id="shape159-619" v:mID="159" v:groupContext="shape">
+ <title>Rectangle.159</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow159-620" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="193.823" width="271.116" height="148.722" class="st58"/>
+ </g>
+ <rect x="0" y="193.823" width="271.116" height="148.722" class="st59"/>
+ </g>
+ <g id="shape160-623" v:mID="160" v:groupContext="shape" transform="translate(271.116,0)">
+ <title>Rectangle.160</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow160-624" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="193.823" width="202.104" height="148.722" class="st58"/>
+ </g>
+ <rect x="0" y="193.823" width="202.104" height="148.722" class="st59"/>
+ </g>
+ <g id="shape161-627" v:mID="161" v:groupContext="shape" transform="translate(83.578,-151.241)">
+ <title>Sheet.161</title>
+ <desc>(a) Distributed Web Cache</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54.3546" cy="333.806" width="108.71" height="17.4792"/>
+ <g id="shadow161-628" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="325.066" width="108.709" height="17.4792" class="st9"/>
+ </g>
+ <rect x="0" y="325.066" width="108.709" height="17.4792" class="st10"/>
+ <text x="4" y="336.81" class="st60" v:langID="1033"><v:paragraph v:spLine="-1.5" v:spBefore="8" v:spAfter="16"
+ v:bulletSize="0.166667"/><v:tabList/>(a) Distributed Web Cache</text> </g>
+ <g id="shape162-632" v:mID="162" v:groupContext="shape" transform="translate(319.513,-151.241)">
+ <title>Sheet.162</title>
+ <desc>(b) Detecting Routing Loops</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54.3546" cy="333.806" width="108.71" height="17.4792"/>
+ <g id="shadow162-633" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="325.066" width="108.709" height="17.4792" class="st9"/>
+ </g>
+ <rect x="0" y="325.066" width="108.709" height="17.4792" class="st10"/>
+ <text x="4" y="336.81" class="st60" v:langID="1033"><v:paragraph v:spLine="-1.5" v:spBefore="8" v:spAfter="16"
+ v:bulletSize="0.166667"/><v:tabList/>(b) Detecting Routing Loops</text> </g>
+ <g id="shape163-637" v:mID="163" v:groupContext="shape" transform="translate(77.5283,-3.35965)">
+ <title>Sheet.163</title>
+ <desc>(c) In-order Workload Scheduler</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="63.5211" cy="333.806" width="127.05" height="17.4792"/>
+ <g id="shadow163-638" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="325.066" width="127.042" height="17.4792" class="st9"/>
+ </g>
+ <rect x="0" y="325.066" width="127.042" height="17.4792" class="st10"/>
+ <text x="4" y="336.81" class="st60" v:langID="1033"><v:paragraph v:spLine="-1.5" v:spBefore="8" v:spAfter="16"
+ v:bulletSize="0.166667"/><v:tabList/>(c) In-order Workload Scheduler</text> </g>
+ <g id="shape164-642" v:mID="164" v:groupContext="shape" transform="translate(307.414,-3.35965)">
+ <title>Sheet.164</title>
+ <desc>(d) Database Semi-join Operations</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="66.2253" cy="333.806" width="132.46" height="17.4792"/>
+ <g id="shadow164-643" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="325.066" width="132.451" height="17.4792" class="st9"/>
+ </g>
+ <rect x="0" y="325.066" width="132.451" height="17.4792" class="st10"/>
+ <text x="4" y="336.81" class="st60" v:langID="1033"><v:paragraph v:spLine="-1.5" v:spBefore="8" v:spAfter="16"
+ v:bulletSize="0.166667"/><v:tabList/>(d) Database Semi-join Operations</text> </g>
+ </g>
+ </g>
+</svg>
diff --git a/doc/guides/prog_guide/img/member_i2.svg b/doc/guides/prog_guide/img/member_i2.svg
new file mode 100644
index 0000000..759c654
--- /dev/null
+++ b/doc/guides/prog_guide/img/member_i2.svg
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by Microsoft Visio, SVG Export memship_i2.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+ xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="4.38194in" height="1.25694in"
+ viewBox="0 0 315.5 90.5" xml:space="preserve" color-interpolation-filters="sRGB" class="st6">
+ <v:documentProperties v:langID="1033" v:viewMarkup="false"/>
+
+ <style type="text/css">
+ <![CDATA[
+ .st1 {fill:none;stroke:none;stroke-width:0.25}
+ .st2 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em}
+ .st3 {baseline-shift:32.4943%;font-size:0.649886em}
+ .st4 {font-size:1em}
+ .st5 {font-family:Cambria Math;font-size:1em}
+ .st6 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+ ]]>
+ </style>
+
+ <g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+ <title>Page-1</title>
+ <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+ <g id="shape3-1" v:mID="3" v:groupContext="shape" transform="translate(0.25,-0.25)">
+ <title>Sheet.3</title>
+ <desc>False Positive Probability = (1-(1-1/m)kn)k ≃ (1-ekn/m)k</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="157.5" cy="45.5" width="315" height="90"/>
+ <rect x="0" y="0.5" width="315" height="90" class="st1"/>
+ <text x="8.28" y="49.82" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>False Positive Probability = (1-(1-1/m)<tspan
+ dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan dy="0.152em" class="st4">)</tspan><tspan
+ dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan><tspan dy="0.152em" class="st4"> </tspan><tspan
+ class="st5">≃</tspan> (1-e<tspan dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan class="st3"
+ v:baseFontSize="14">/</tspan><tspan class="st3" v:baseFontSize="14">m</tspan><tspan dy="0.152em"
+ class="st4">)</tspan><tspan dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan></text> </g>
+ </g>
+</svg>
diff --git a/doc/guides/prog_guide/img/member_i3.svg b/doc/guides/prog_guide/img/member_i3.svg
new file mode 100644
index 0000000..41e92cb
--- /dev/null
+++ b/doc/guides/prog_guide/img/member_i3.svg
@@ -0,0 +1,148 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by Microsoft Visio, SVG Export memship_i3.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+ width="4.71875in" height="2.84375in" viewBox="0 0 339.75 204.75" xml:space="preserve" color-interpolation-filters="sRGB"
+ class="st14">
+ <style type="text/css">
+ <![CDATA[
+ .st1 {visibility:visible}
+ .st2 {fill:#5b9bd5;fill-opacity:0.22;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.22}
+ .st3 {fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25}
+ .st4 {marker-end:url(#mrkr5-32);stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st5 {fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-opacity:1;stroke-width:0.28409090909091}
+ .st6 {fill:#feffff;font-family:Calibri;font-size:0.833336em}
+ .st7 {font-size:1em}
+ .st8 {fill:#deebf6;stroke:#c7c8c8;stroke-width:0.25}
+ .st9 {fill:#000000;font-family:Calibri;font-size:0.833336em}
+ .st10 {marker-end:url(#mrkr5-84);stroke:#ff0000;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st11 {fill:#ff0000;fill-opacity:1;stroke:#ff0000;stroke-opacity:1;stroke-width:0.28409090909091}
+ .st12 {fill:none;stroke:none;stroke-width:0.25}
+ .st13 {fill:#ff0000;font-family:Calibri;font-size:1.00001em}
+ .st14 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+ ]]>
+ </style>
+
+ <defs id="Markers">
+ <g id="lend5">
+ <path d="M 2 1 L 0 0 L 1.98117 -0.993387 C 1.67173 -0.364515 1.67301 0.372641 1.98465 1.00043 " style="stroke:none"/>
+ </g>
+ <marker id="mrkr5-32" class="st5" refX="-6.16" orient="auto" markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-3.52,-3.52) "/>
+ </marker>
+ <marker id="mrkr5-84" class="st11" refX="-5.8" orient="auto" markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-3.52,-3.52) "/>
+ </marker>
+ </defs>
+ <defs id="Filters">
+ <filter id="filter_2">
+ <feGaussianBlur stdDeviation="2"/>
+ </filter>
+ </defs>
+ <g>
+ <title>Page-1</title>
+ <g id="group174-1" transform="translate(3.0294,-5.3478)">
+ <title>Sheet.174</title>
+ <g id="shape155-2" transform="translate(99,-99)">
+ <title>Circle</title>
+ <g id="shadow155-3" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st2"/>
+ </g>
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st3"/>
+ </g>
+ <g id="shape156-7" transform="translate(207,-108)">
+ <title>Circle.156</title>
+ <g id="shadow156-8" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st2"/>
+ </g>
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st3"/>
+ </g>
+ <g id="shape157-12" transform="translate(153,-162)">
+ <title>Circle.157</title>
+ <g id="shadow157-13" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st2"/>
+ </g>
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st3"/>
+ </g>
+ <g id="shape158-17" transform="translate(297,-99)">
+ <title>Circle.158</title>
+ <g id="shadow158-18" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st2"/>
+ </g>
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st3"/>
+ </g>
+ <g id="shape159-22" transform="translate(180,-36)">
+ <title>Circle.159</title>
+ <g id="shadow159-23" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st2"/>
+ </g>
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st3"/>
+ </g>
+ <g id="shape160-27" transform="translate(109.604,-115.419) rotate(-7.12502)">
+ <title>Sheet.160</title>
+ <path d="M0 204.75 L66.4 204.75" class="st4"/>
+ </g>
+ <g id="shape161-33" transform="translate(276.661,-123.214) rotate(9.46232)">
+ <title>Sheet.161</title>
+ <path d="M0 204.75 L48.58 204.75" class="st4"/>
+ </g>
+ <g id="shape162-38" transform="translate(246.135,262.572) rotate(-160.346)">
+ <title>Sheet.162</title>
+ <path d="M0 204.75 L127.63 204.75" class="st4"/>
+ </g>
+ <g id="shape163-43" transform="translate(284.391,198.775) rotate(141.977)">
+ <title>Sheet.163</title>
+ <path d="M0 204.75 L46.23 204.75" class="st4"/>
+ </g>
+ <g id="shape164-48" transform="translate(70.6118,307.655) rotate(-145.945)">
+ <title>Sheet.164</title>
+ <path d="M0 204.75 L60.88 204.75" class="st4"/>
+ </g>
+ <g id="shape167-53" transform="translate(72,0)">
+ <title>Rectangle.167</title>
+ <desc>BF of IDs</desc>
+ <g id="shadow167-54" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <rect x="0" y="177.75" width="36" height="27" class="st2"/>
+ </g>
+ <rect x="0" y="177.75" width="36" height="27" class="st3"/>
+ <text x="7.69" y="188.25" class="st6">BF of <tspan x="11.71" dy="1.2em" class="st7">IDs</tspan></text> </g>
+ <g id="shape168-60" transform="translate(108,0)">
+ <title>Rectangle.168</title>
+ <desc>Packet</desc>
+ <g id="shadow168-61" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <rect x="0" y="177.75" width="90" height="27" class="st2"/>
+ </g>
+ <rect x="0" y="177.75" width="90" height="27" class="st8"/>
+ <text x="31.47" y="194.25" class="st9">Packet</text> </g>
+ <g id="shape169-66" transform="translate(0,-63)">
+ <title>Rectangle.169</title>
+ <desc>BF of IDs</desc>
+ <g id="shadow169-67" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <rect x="0" y="177.75" width="36" height="27" class="st2"/>
+ </g>
+ <rect x="0" y="177.75" width="36" height="27" class="st3"/>
+ <text x="7.69" y="188.25" class="st6">BF of <tspan x="11.71" dy="1.2em" class="st7">IDs</tspan></text> </g>
+ <g id="shape170-73" transform="translate(36,-63)">
+ <title>Rectangle.170</title>
+ <desc>Packet</desc>
+ <g id="shadow170-74" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <rect x="0" y="177.75" width="90" height="27" class="st2"/>
+ </g>
+ <rect x="0" y="177.75" width="90" height="27" class="st8"/>
+ <text x="31.47" y="194.25" class="st9">Packet</text> </g>
+ <g id="shape171-79" transform="translate(240.248,331.493) rotate(161.565)">
+ <title>Sheet.171</title>
+ <path d="M-0 190.52 A81.3416 36.0611 -153.48 0 0 82.31 195.86 L82.49 195.55" class="st10"/>
+ </g>
+ <g id="shape172-85" transform="translate(156.426,260.029) rotate(161.565)">
+ <title>Sheet.172</title>
+ <path d="M-0 181.6 A88.1422 54.1439 -124.1 0 0 82.68 187.13 L82.83 186.81" class="st10"/>
+ </g>
+ <g id="shape173-90" transform="translate(18,-121.5)">
+ <title>Sheet.173</title>
+ <desc>Encode ID</desc>
+ <rect x="0" y="177.75" width="63" height="27" class="st12"/>
+ <text x="7.02" y="194.85" class="st13">Encode ID</text> </g>
+ </g>
+ </g>
+</svg>
diff --git a/doc/guides/prog_guide/img/member_i4.svg b/doc/guides/prog_guide/img/member_i4.svg
new file mode 100644
index 0000000..a2b6f2f
--- /dev/null
+++ b/doc/guides/prog_guide/img/member_i4.svg
@@ -0,0 +1,450 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.0//EN" "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd">
+<!-- Generated by Microsoft Visio 11.0, SVG Export, v1.0 memship_i4.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+ xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="7.625in" height="3.125in" viewBox="0 0 549 225"
+ xml:space="preserve" color-interpolation-filters="sRGB" class="st18">
+ <v:documentProperties v:langID="1033" v:viewMarkup="false"/>
+
+ <style type="text/css">
+ <![CDATA[
+ .st1 {fill:none;stroke:#ff0000;stroke-width:0.25}
+ .st2 {fill:#5b9bd5;font-family:Calibri;font-size:1.00001em}
+ .st3 {marker-end:url(#mrkr5-10);stroke:#ff0000;stroke-linecap:round;stroke-linejoin:round;stroke-width:2.25}
+ .st4 {fill:#ff0000;fill-opacity:1;stroke:#ff0000;stroke-opacity:1;stroke-width:0.47169811320755}
+ .st5 {visibility:visible}
+ .st6 {fill:#5b9bd5;fill-opacity:0.22;stroke:#5b9bd5;stroke-opacity:0.22;stroke-width:0.25}
+ .st7 {fill:#5b9bd5;stroke:#c8c8c8;stroke-width:0.25}
+ .st8 {fill:none;stroke:none;stroke-width:0.25}
+ .st9 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em}
+ .st10 {stroke:#5b9bd5;stroke-dasharray:1.5,3;stroke-linecap:round;stroke-linejoin:round;stroke-width:1.5}
+ .st11 {fill:#5b9bd5;fill-opacity:0.25;stroke:#5b9bd5;stroke-opacity:0.25;stroke-width:0.75}
+ .st12 {fill:#4f88bb;stroke:#41719c;stroke-width:0.75}
+ .st13 {fill:#ffffff;font-family:Calibri;font-size:1.00001em;font-weight:bold}
+ .st14 {fill:#000000;font-family:Calibri;font-size:1.00001em}
+ .st15 {font-size:1em}
+ .st16 {marker-end:url(#mrkr5-162);stroke:#000000;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st17 {fill:#000000;fill-opacity:1;stroke:#000000;stroke-opacity:1;stroke-width:0.28409090909091}
+ .st18 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+ ]]>
+ </style>
+
+ <defs id="Markers">
+ <g id="lend5">
+ <path d="M 2 1 L 0 0 L 1.98117 -0.993387 C 1.67173 -0.364515 1.67301 0.372641 1.98465 1.00043 " style="stroke:none"/>
+ </g>
+ <marker id="mrkr5-10" class="st4" v:arrowType="5" v:arrowSize="2" v:setback="3.71" refX="-3.71" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-2.12,-2.12) "/>
+ </marker>
+ <marker id="mrkr5-162" class="st17" v:arrowType="5" v:arrowSize="2" v:setback="5.8" refX="-5.8" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-3.52,-3.52) "/>
+ </marker>
+ </defs>
+ <g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+ <title>Page-1</title>
+ <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+ <g id="group47-1" transform="translate(3.0294,-0.25)" v:mID="47" v:groupContext="group">
+ <title>Sheet.47</title>
+ <g id="shape1-2" v:mID="1" v:groupContext="shape" transform="translate(177.75,-191.922)">
+ <title>Sheet.1</title>
+ <desc>Element</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="216" width="108" height="18"/>
+ <rect x="0" y="207" width="108" height="18" class="st1"/>
+ <text x="33.77" y="219.6" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Element</text> </g>
+ <g id="shape2-5" v:mID="2" v:groupContext="shape" transform="translate(456.75,33.0781) rotate(90)">
+ <title>Sheet.2</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 225 L18.65 225" class="st3"/>
+ </g>
+ <g id="shape3-11" v:mID="3" v:groupContext="shape" transform="translate(0,-67.0469)">
+ <title>Rectangle.54</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow3-12" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape4-15" v:mID="4" v:groupContext="shape" transform="translate(27,-67.0469)">
+ <title>Rectangle.55</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow4-16" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape5-19" v:mID="5" v:groupContext="shape" transform="translate(54,-67.0469)">
+ <title>Rectangle.56</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow5-20" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape6-23" v:mID="6" v:groupContext="shape" transform="translate(0,-53.5469)">
+ <title>Rectangle.57</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow6-24" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape7-27" v:mID="7" v:groupContext="shape" transform="translate(27,-53.5469)">
+ <title>Rectangle.58</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow7-28" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape8-31" v:mID="8" v:groupContext="shape" transform="translate(54,-53.5469)">
+ <title>Rectangle.59</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow8-32" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape9-35" v:mID="9" v:groupContext="shape" transform="translate(5.625,-72.6719)">
+ <title>Sheet.9</title>
+ <desc>BF-1</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="36" cy="211.5" width="72" height="27"/>
+ <rect x="0" y="198" width="72" height="27" class="st8"/>
+ <text x="23.29" y="215.7" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>BF-1</text> </g>
+ <g id="shape10-38" v:mID="10" v:groupContext="shape" transform="translate(128.25,-65.0781)">
+ <title>Rectangle.74</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow10-39" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape11-42" v:mID="11" v:groupContext="shape" transform="translate(155.25,-65.0781)">
+ <title>Rectangle.75</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow11-43" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape12-46" v:mID="12" v:groupContext="shape" transform="translate(182.25,-65.0781)">
+ <title>Rectangle.76</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow12-47" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape13-50" v:mID="13" v:groupContext="shape" transform="translate(128.25,-51.5781)">
+ <title>Rectangle.77</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow13-51" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape14-54" v:mID="14" v:groupContext="shape" transform="translate(155.25,-51.5781)">
+ <title>Rectangle.78</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow14-55" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape15-58" v:mID="15" v:groupContext="shape" transform="translate(182.25,-51.5781)">
+ <title>Rectangle.79</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow15-59" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape16-62" v:mID="16" v:groupContext="shape" transform="translate(301.5,-65.0781)">
+ <title>Rectangle.81</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow16-63" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape17-66" v:mID="17" v:groupContext="shape" transform="translate(328.5,-65.0781)">
+ <title>Rectangle.82</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow17-67" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape18-70" v:mID="18" v:groupContext="shape" transform="translate(355.5,-65.0781)">
+ <title>Rectangle.83</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow18-71" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape19-74" v:mID="19" v:groupContext="shape" transform="translate(301.5,-51.5781)">
+ <title>Rectangle.84</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow19-75" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape20-78" v:mID="20" v:groupContext="shape" transform="translate(328.5,-51.5781)">
+ <title>Rectangle.85</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow20-79" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape21-82" v:mID="21" v:groupContext="shape" transform="translate(355.5,-51.5781)">
+ <title>Rectangle.86</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow21-83" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape22-86" v:mID="22" v:groupContext="shape" transform="translate(447.75,-65.6406)">
+ <title>Rectangle.88</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow22-87" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape23-90" v:mID="23" v:groupContext="shape" transform="translate(474.75,-65.6406)">
+ <title>Rectangle.89</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow23-91" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape24-94" v:mID="24" v:groupContext="shape" transform="translate(501.75,-65.6406)">
+ <title>Rectangle.90</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow24-95" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape25-98" v:mID="25" v:groupContext="shape" transform="translate(447.75,-52.1406)">
+ <title>Rectangle.91</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow25-99" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape26-102" v:mID="26" v:groupContext="shape" transform="translate(474.75,-52.1406)">
+ <title>Rectangle.92</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow26-103" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape27-106" v:mID="27" v:groupContext="shape" transform="translate(501.75,-52.1406)">
+ <title>Rectangle.93</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow27-107" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape28-110" v:mID="28" v:groupContext="shape" transform="translate(213.75,-63.9531)">
+ <title>Sheet.28</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 225 L83.25 225" class="st10"/>
+ </g>
+ <g id="shape29-113" v:mID="29" v:groupContext="shape" transform="translate(387,-63.9531)">
+ <title>Sheet.29</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 225 L54 225" class="st10"/>
+ </g>
+ <g id="group31-116" transform="translate(184.5,-113.172)" v:mID="31" v:groupContext="group">
+ <title>Sheet.31</title>
+ <g id="shape32-117" v:mID="32" v:groupContext="shape" transform="translate(225,173.25) rotate(90)">
+ <title>Block Arrow</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow32-118" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,1.9728,-0.3456)" class="st5">
+ <path d="M0 225 L25.87 225 L51.75 177.75 L25.87 130.5 L0 130.5 L0 225 Z" class="st11"/>
+ </g>
+ <path d="M0 225 L25.87 225 L51.75 177.75 L25.87 130.5 L0 130.5 L0 225 Z" class="st12"/>
+ </g>
+ <g id="shape33-121" v:mID="33" v:groupContext="shape" transform="translate(2.25,-24.3529)">
+ <title>Sheet.33</title>
+ <desc>h1, h2 .. hk</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="45" cy="215.868" width="90" height="18.2647"/>
+ <rect x="0" y="206.735" width="90" height="18.2647" class="st8"/>
+ <text x="17.56" y="219.47" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>h1, h2 .. hk</text> </g>
+ </g>
+ <g id="shape34-124" v:mID="34" v:groupContext="shape" transform="translate(307.011,286.73) rotate(152.323)">
+ <title>Sheet.34</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 225 L128.85 225" class="st3"/>
+ </g>
+ <g id="shape35-129" v:mID="35" v:groupContext="shape" transform="translate(433.272,125.452) rotate(99.7172)">
+ <title>Sheet.35</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 225 L58.31 225" class="st3"/>
+ </g>
+ <g id="shape36-134" v:mID="36" v:groupContext="shape" transform="translate(407.724,-64.1459) rotate(45)">
+ <title>Sheet.36</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 225 L79.16 225" class="st3"/>
+ </g>
+ <g id="shape37-139" v:mID="37" v:groupContext="shape" transform="translate(320.441,-127.12) rotate(15.6155)">
+ <title>Sheet.37</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 225 L200.75 225" class="st3"/>
+ </g>
+ <g id="shape38-144" v:mID="38" v:groupContext="shape" transform="translate(132.75,-75.2588)">
+ <title>Sheet.38</title>
+ <desc>BF-2</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="36" cy="211.5" width="72" height="27"/>
+ <rect x="0" y="198" width="72" height="27" class="st8"/>
+ <text x="23.29" y="215.7" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>BF-2</text> </g>
+ <g id="shape39-147" v:mID="39" v:groupContext="shape" transform="translate(303.75,-70.7588)">
+ <title>Sheet.39</title>
+ <desc>BF-X</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="36" cy="211.5" width="72" height="27"/>
+ <rect x="0" y="198" width="72" height="27" class="st8"/>
+ <text x="23.2" y="215.7" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>BF-X</text> </g>
+ <g id="shape40-150" v:mID="40" v:groupContext="shape" transform="translate(447.75,-75.2588)">
+ <title>Sheet.40</title>
+ <desc>BF-L</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="36" cy="211.5" width="72" height="27"/>
+ <rect x="0" y="198" width="72" height="27" class="st8"/>
+ <text x="23.89" y="215.7" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>BF-L</text> </g>
+ <g id="shape41-153" v:mID="41" v:groupContext="shape" transform="translate(300.375,-117)">
+ <title>Sheet.41</title>
+ <desc>Hashing for lookup/Insertion into a vector of BFs happens once</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="90" cy="202.5" width="180" height="45"/>
+ <rect x="0" y="180" width="180" height="45" class="st8"/>
+ <text x="4.6" y="198.9" class="st14" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Hashing for lookup/Insertion into a <tspan
+ x="23.06" dy="1.2em" class="st15">vector of BFs happens once</tspan></text> </g>
+ <g id="shape44-157" v:mID="44" v:groupContext="shape" transform="translate(249.698,-151.505) rotate(-3.74012)">
+ <title>Sheet.44</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M-0 225 A93.4958 45.6256 42.23 0 1 79.38 221.66 L79.68 221.85" class="st16"/>
+ </g>
+ <g id="shape45-163" v:mID="45" v:groupContext="shape" transform="translate(30.375,0.25)">
+ <title>Sheet.45</title>
+ <desc>Lookup/Insertion is done in the series of BFs, one by one or ...</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="233.048" cy="202.5" width="466.1" height="45"/>
+ <rect x="0" y="180" width="466.096" height="45" class="st8"/>
+ <text x="4.34" y="206.1" class="st14" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Lookup/Insertion is done in the series of BFs, one by one or can be optimized to do in parallel. </text> </g>
+ <g id="shape46-166" v:mID="46" v:groupContext="shape" transform="translate(123.252,-43.6868) rotate(17.0249)">
+ <title>Sheet.46</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M-0 225 A88.2185 43.0621 47.63 0 1 70.31 221.39 L70.6 221.6" class="st16"/>
+ </g>
+ </g>
+ </g>
+</svg>
diff --git a/doc/guides/prog_guide/img/member_i5.svg b/doc/guides/prog_guide/img/member_i5.svg
new file mode 100644
index 0000000..c1728cf
--- /dev/null
+++ b/doc/guides/prog_guide/img/member_i5.svg
@@ -0,0 +1,163 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.0//EN" "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd">
+<!-- Generated by Microsoft Visio 11.0, SVG Export, v1.0 memship_i5.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+ xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="5.30481in" height="1.96146in"
+ viewBox="0 0 381.946 141.225" xml:space="preserve" color-interpolation-filters="sRGB" class="st15">
+ <v:documentProperties v:langID="1033" v:viewMarkup="false"/>
+
+ <style type="text/css">
+ <![CDATA[
+ .st1 {visibility:visible}
+ .st2 {fill:#5b9bd5;fill-opacity:0.22;stroke:#5b9bd5;stroke-opacity:0.22;stroke-width:0.25}
+ .st3 {fill:#5b9bd5;stroke:#c8c8c8;stroke-width:0.25}
+ .st4 {fill:none;stroke:none;stroke-width:0.25}
+ .st5 {fill:#ffffff;font-family:Calibri;font-size:1.16666em;font-weight:bold}
+ .st6 {marker-end:url(#mrkr5-14);stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st7 {fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-opacity:1;stroke-width:0.28409090909091}
+ .st8 {fill:#5b9bd5;font-family:Calibri;font-size:0.833336em}
+ .st9 {fill:#5b9bd5;font-family:Calibri;font-size:0.75em}
+ .st10 {stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st11 {marker-end:url(#mrkr5-63);stroke:#000000;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.75}
+ .st12 {fill:#000000;fill-opacity:1;stroke:#000000;stroke-opacity:1;stroke-width:0.22935779816514}
+ .st13 {fill:#000000;font-family:Calibri;font-size:1.00001em}
+ .st14 {font-size:1em}
+ .st15 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+ ]]>
+ </style>
+
+ <defs id="Markers">
+ <g id="lend5">
+ <path d="M 2 1 L 0 0 L 1.98117 -0.993387 C 1.67173 -0.364515 1.67301 0.372641 1.98465 1.00043 " style="stroke:none"/>
+ </g>
+ <marker id="mrkr5-14" class="st7" v:arrowType="5" v:arrowSize="2" v:setback="6.16" refX="-6.16" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-3.52,-3.52) "/>
+ </marker>
+ <marker id="mrkr5-63" class="st12" v:arrowType="5" v:arrowSize="2" v:setback="7.15" refX="-7.15" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-4.36,-4.36) "/>
+ </marker>
+ </defs>
+ <g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+ <title>Page-1</title>
+ <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+ <g id="group1-1" transform="translate(191.995,-19.4751)" v:mID="1" v:groupContext="group">
+ <title>Sheet.1</title>
+ <g id="shape2-2" v:mID="2" v:groupContext="shape" transform="translate(146.944,42.2251) rotate(90)">
+ <title>Triangle</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow2-3" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,1.9728,-0.3456)" class="st1">
+ <path d="M99 141.23 L49.5 58.62 L0 141.23 L99 141.23 Z" class="st2"/>
+ </g>
+ <path d="M99 141.23 L49.5 58.62 L0 141.23 L99 141.23 Z" class="st3"/>
+ </g>
+ <g id="shape3-6" v:mID="3" v:groupContext="shape" transform="translate(0,-34.65)">
+ <title>Sheet.3</title>
+ <desc>vBF</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="38.1251" cy="126.375" width="76.26" height="29.7"/>
+ <rect x="0" y="111.525" width="76.2502" height="29.7" class="st4"/>
+ <text x="27.68" y="130.58" class="st5" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>vBF </text> </g>
+ </g>
+ <g id="shape4-9" v:mID="4" v:groupContext="shape" transform="translate(126.724,-100.475)">
+ <title>Sheet.4</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 141.23 L64.83 141.23" class="st6"/>
+ </g>
+ <g id="shape5-15" v:mID="5" v:groupContext="shape" transform="translate(103.5,-101.775)">
+ <title>Sheet.5</title>
+ <desc>Flow Key</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="39.9122" cy="135.863" width="79.83" height="10.7251"/>
+ <rect x="0" y="130.5" width="79.8244" height="10.7251" class="st4"/>
+ <text x="21.78" y="138.86" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow Key</text> </g>
+ <g id="shape6-18" v:mID="6" v:groupContext="shape" transform="translate(221.726,-56.2468) rotate(-24.5123)">
+ <title>Sheet.6</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 141.23 L65.42 141.23" class="st6"/>
+ </g>
+ <g id="shape7-23" v:mID="7" v:groupContext="shape" transform="translate(280.318,-68.9751)">
+ <title>Sheet.7</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 141.23 L64.83 141.23" class="st6"/>
+ </g>
+ <g id="shape8-28" v:mID="8" v:groupContext="shape" transform="translate(338.125,-56.6022) rotate(24.1625)">
+ <title>Sheet.8</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 141.23 L70.8 141.23" class="st6"/>
+ </g>
+ <g id="shape9-33" v:mID="9" v:groupContext="shape" transform="translate(197.714,217.975) rotate(180)">
+ <title>Sheet.9</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 141.23 L51.03 141.23" class="st6"/>
+ </g>
+ <g id="shape10-38" v:mID="10" v:groupContext="shape" transform="translate(18,-67.5)">
+ <title>Sheet.10</title>
+ <desc>New Flow =&#62; New Assignment</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="80.4201" cy="134.475" width="160.85" height="13.5"/>
+ <rect x="0" y="127.725" width="160.84" height="13.5" class="st4"/>
+ <text x="25.11" y="137.18" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>New Flow =&#62; New Assignment</text> </g>
+ <g id="shape11-41" v:mID="11" v:groupContext="shape" transform="translate(198.032,253.975) rotate(180)">
+ <title>Sheet.11</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 141.23 L51.03 141.23" class="st6"/>
+ </g>
+ <g id="shape12-46" v:mID="12" v:groupContext="shape" transform="translate(0,-31.5)">
+ <title>Sheet.12</title>
+ <desc>Old Flow =&#62; forward to specific thread</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="81" cy="136.725" width="162.01" height="9"/>
+ <rect x="0" y="132.225" width="162" height="9" class="st4"/>
+ <text x="11.04" y="139.43" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Old Flow =&#62; forward to specific thread</text> </g>
+ <g id="shape13-49" v:mID="13" v:groupContext="shape" transform="translate(494.552,22.75) rotate(90)">
+ <title>Sheet.13</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 134.49 C3.18 142.89 7.57 142.28 11.25 138.99 C15.79 134.93 19.26 126.78 27 134.49" class="st10"/>
+ </g>
+ <g id="shape14-52" v:mID="14" v:groupContext="shape" transform="translate(494.552,58.75) rotate(90)">
+ <title>Sheet.14</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 134.49 C3.18 142.89 7.57 142.28 11.25 138.99 C15.79 134.93 19.26 126.78 27 134.49" class="st10"/>
+ </g>
+ <g id="shape15-55" v:mID="15" v:groupContext="shape" transform="translate(494.552,94.75) rotate(90)">
+ <title>Sheet.15</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 134.49 C3.18 142.89 7.57 142.28 11.25 138.99 C15.79 134.93 19.26 126.78 27 134.49" class="st10"/>
+ </g>
+ <g id="shape17-58" v:mID="17" v:groupContext="shape" transform="translate(348.769,-25.0593) rotate(44.5185)">
+ <title>Sheet.17</title>
+ <path d="M-0 141.23 A35.1884 19.2595 167.75 0 1 42.43 138.27 L42.74 138.46" class="st11"/>
+ </g>
+ <g id="shape18-64" v:mID="18" v:groupContext="shape" transform="translate(222.188,-5.40005)">
+ <title>Sheet.18</title>
+ <desc>A BF corresponding to each worker thread</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="59.0625" cy="127.725" width="118.13" height="27"/>
+ <rect x="0" y="114.225" width="118.125" height="27" class="st4"/>
+ <text x="5.14" y="124.13" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>A BF corresponding to <tspan
+ x="11.19" dy="1.2em" class="st14">each worker thread</tspan></text> </g>
+ </g>
+</svg>
diff --git a/doc/guides/prog_guide/img/member_i6.svg b/doc/guides/prog_guide/img/member_i6.svg
new file mode 100644
index 0000000..265179f
--- /dev/null
+++ b/doc/guides/prog_guide/img/member_i6.svg
@@ -0,0 +1,332 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.0//EN" "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd">
+<!-- Generated by Microsoft Visio 11.0, SVG Export, v1.0 memship_i6.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+ xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="8in" height="3.625in" viewBox="0 0 576 261"
+ xml:space="preserve" color-interpolation-filters="sRGB" class="st16">
+ <v:documentProperties v:langID="1033" v:viewMarkup="false"/>
+
+ <style type="text/css">
+ <![CDATA[
+ .st1 {visibility:visible}
+ .st2 {fill:#5b9bd5;fill-opacity:0.22;stroke:#5b9bd5;stroke-opacity:0.22;stroke-width:0.25}
+ .st3 {fill:#5b9bd5;stroke:#c8c8c8;stroke-width:0.25}
+ .st4 {fill:#feffff;font-family:Calibri;font-size:0.666664em;font-weight:bold}
+ .st5 {font-size:1em}
+ .st6 {fill:#70ad47;fill-opacity:0.5;stroke:#00b050;stroke-width:1.5}
+ .st7 {fill:none;stroke:none;stroke-width:0.25}
+ .st8 {fill:#00b050;font-family:Calibri;font-size:1.00001em}
+ .st9 {fill:none;stroke:#ff0000;stroke-width:0.25}
+ .st10 {fill:#5b9bd5;font-family:Calibri;font-size:0.833336em}
+ .st11 {marker-end:url(#mrkr5-29);stroke:#ff0000;stroke-linecap:round;stroke-linejoin:round;stroke-width:2.25}
+ .st12 {fill:#ff0000;fill-opacity:1;stroke:#ff0000;stroke-opacity:1;stroke-width:0.47169811320755}
+ .st13 {fill:#5b9bd5;font-family:Calibri;font-size:0.75em}
+ .st14 {fill:#92d050;stroke:#c8c8c8;stroke-width:0.25}
+ .st15 {fill:#5b9bd5;font-family:Calibri;font-size:1.00001em;font-weight:bold}
+ .st16 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+ ]]>
+ </style>
+
+ <defs id="Markers">
+ <g id="lend5">
+ <path d="M 2 1 L 0 0 L 1.98117 -0.993387 C 1.67173 -0.364515 1.67301 0.372641 1.98465 1.00043 " style="stroke:none"/>
+ </g>
+ <marker id="mrkr5-29" class="st12" v:arrowType="5" v:arrowSize="2" v:setback="3.71" refX="-3.71" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-2.12,-2.12) "/>
+ </marker>
+ </defs>
+ <g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+ <title>Page-1</title>
+ <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+ <g id="group121-1" transform="translate(21.0294,-9.8478)" v:mID="121" v:groupContext="group">
+ <title>Sheet.121</title>
+ <g id="shape49-2" v:mID="49" v:groupContext="shape" transform="translate(396.989,-54.9268)">
+ <title>Rectangle.2</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow49-3" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="219.549" width="99.4817" height="41.4507" class="st2"/>
+ </g>
+ <rect x="0" y="219.549" width="99.4817" height="41.4507" class="st3"/>
+ </g>
+ <g id="shape50-6" v:mID="50" v:groupContext="shape" transform="translate(248.261,-12.1936)">
+ <title>Rectangle.4</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow50-7" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="178.099" width="99.4817" height="82.9014" class="st2"/>
+ </g>
+ <rect x="0" y="178.099" width="99.4817" height="82.9014" class="st3"/>
+ </g>
+ <g id="shape52-10" v:mID="52" v:groupContext="shape" transform="translate(6.07514E-013,-29.0155)">
+ <title>Rectangle.10</title>
+ <desc>Signatures for target 1</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="49.7409" cy="225.767" width="99.49" height="70.4662"/>
+ <g id="shadow52-11" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="190.534" width="99.4817" height="70.4662" class="st2"/>
+ </g>
+ <rect x="0" y="190.534" width="99.4817" height="70.4662" class="st3"/>
+ <text x="26.54" y="223.37" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Signatures for<v:newlineChar/><tspan
+ x="36.73" dy="1.2em" class="st5">target </tspan>1</text> </g>
+ <g id="shape53-16" v:mID="53" v:groupContext="shape" transform="translate(239.971,-20.4837)">
+ <title>Sheet.53</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <rect x="0" y="248.565" width="116.062" height="12.4352" class="st6"/>
+ </g>
+ <g id="shape54-18" v:mID="54" v:groupContext="shape" transform="translate(353.649,-19.9346)">
+ <title>Sheet.54</title>
+ <desc>Match 1</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="33.1606" cy="254.782" width="66.33" height="12.4352"/>
+ <rect x="0" y="248.565" width="66.3211" height="12.4352" class="st7"/>
+ <text x="13.06" y="258.38" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Match 1</text> </g>
+ <g id="shape55-21" v:mID="55" v:groupContext="shape" transform="translate(216.989,-210.652)">
+ <title>Sheet.55</title>
+ <desc>Packet Payload</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="49.7409" cy="252.71" width="99.49" height="16.5803"/>
+ <rect x="0" y="244.42" width="99.4817" height="16.5803" class="st9"/>
+ <text x="19.04" y="255.71" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Packet Payload</text> </g>
+ <g id="shape56-24" v:mID="56" v:groupContext="shape" transform="translate(526.665,52.2365) rotate(90)">
+ <title>Sheet.56</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 261 L16.52 261" class="st11"/>
+ </g>
+ <g id="shape96-30" v:mID="96" v:groupContext="shape" transform="translate(-3.0294,-95.7818)">
+ <title>Sheet.96</title>
+ <desc>Attack Signature Length 1</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="51.75" cy="248.565" width="103.5" height="24.8704"/>
+ <rect x="0" y="236.13" width="103.5" height="24.8704" class="st7"/>
+ <text x="4.79" y="251.26" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Attack Signature Length 1</text> </g>
+ <g id="group114-33" transform="translate(228.359,-134.152)" v:mID="114" v:groupContext="group">
+ <title>Sheet.114</title>
+ <g id="group106-34" transform="translate(0,-24.8704)" v:mID="106" v:groupContext="group">
+ <title>Sheet.106</title>
+ <g id="shape100-35" v:mID="100" v:groupContext="shape" transform="translate(3.65707E-013,-12.4352)">
+ <title>Rectangle.100</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow100-36" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape101-39" v:mID="101" v:groupContext="shape" transform="translate(24.8704,-12.4352)">
+ <title>Rectangle.101</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow101-40" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape102-43" v:mID="102" v:groupContext="shape" transform="translate(49.7409,-12.4352)">
+ <title>Rectangle.102</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow102-44" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape103-47" v:mID="103" v:groupContext="shape">
+ <title>Rectangle.103</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow103-48" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape104-51" v:mID="104" v:groupContext="shape" transform="translate(24.8704,1.13687E-013)">
+ <title>Rectangle.104</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow104-52" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape105-55" v:mID="105" v:groupContext="shape" transform="translate(49.7409,1.13687E-013)">
+ <title>Rectangle.105</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow105-56" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ </g>
+ <g id="group107-59" v:mID="107" v:groupContext="group">
+ <title>Sheet.107</title>
+ <g id="shape108-60" v:mID="108" v:groupContext="shape" transform="translate(3.65707E-013,-12.4352)">
+ <title>Rectangle.100</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow108-61" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape109-64" v:mID="109" v:groupContext="shape" transform="translate(24.8704,-12.4352)">
+ <title>Rectangle.101</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow109-65" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape110-68" v:mID="110" v:groupContext="shape" transform="translate(49.7409,-12.4352)">
+ <title>Rectangle.102</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow110-69" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape111-72" v:mID="111" v:groupContext="shape">
+ <title>Rectangle.103</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <g id="shadow111-73" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st14"/>
+ </g>
+ <g id="shape112-76" v:mID="112" v:groupContext="shape" transform="translate(24.8704,1.13687E-013)">
+ <title>Rectangle.104</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow112-77" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape113-80" v:mID="113" v:groupContext="shape" transform="translate(49.7409,1.13687E-013)">
+ <title>Rectangle.105</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <g id="shadow113-81" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st14"/>
+ </g>
+ </g>
+ </g>
+ <g id="shape89-84" v:mID="89" v:groupContext="shape" transform="translate(398.644,-116.927) rotate(24.4696)">
+ <title>Sheet.89</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 261 L143.75 261" class="st11"/>
+ </g>
+ <g id="shape115-89" v:mID="115" v:groupContext="shape" transform="translate(116.062,-1.19371E-012)">
+ <title>Rectangle.115</title>
+ <desc>Signatures for target 2</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="49.7409" cy="211.259" width="99.49" height="99.4817"/>
+ <g id="shadow115-90" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="161.518" width="99.4817" height="99.4817" class="st2"/>
+ </g>
+ <rect x="0" y="161.518" width="99.4817" height="99.4817" class="st3"/>
+ <text x="26.54" y="208.86" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Signatures for<v:newlineChar/><tspan
+ x="36.73" dy="1.2em" class="st5">target </tspan>2</text> </g>
+ <g id="shape116-95" v:mID="116" v:groupContext="shape" transform="translate(117.989,-95.7818)">
+ <title>Sheet.116</title>
+ <desc>Attack Signature Length 2</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="51.9909" cy="248.565" width="103.99" height="24.8704"/>
+ <rect x="0" y="236.13" width="103.982" height="24.8704" class="st7"/>
+ <text x="5.03" y="251.26" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Attack Signature Length 2</text> </g>
+ <g id="shape118-98" v:mID="118" v:groupContext="shape" transform="translate(392.971,-90.217)">
+ <title>Sheet.118</title>
+ <desc>Attack Signature Length L</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="248.565" width="108" height="24.8704"/>
+ <rect x="0" y="236.13" width="108" height="24.8704" class="st7"/>
+ <text x="7.43" y="251.26" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Attack Signature Length L</text> </g>
+ <g id="shape119-101" v:mID="119" v:groupContext="shape" transform="translate(384.909,-64.9346)">
+ <title>Sheet.119</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <rect x="0" y="248.565" width="116.062" height="12.4352" class="st6"/>
+ </g>
+ <g id="shape120-103" v:mID="120" v:groupContext="shape" transform="translate(491.971,-64.9346)">
+ <title>Sheet.120</title>
+ <desc>Match 2</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="33.1606" cy="254.782" width="66.33" height="12.4352"/>
+ <rect x="0" y="248.565" width="66.3211" height="12.4352" class="st7"/>
+ <text x="13.06" y="258.38" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Match 2</text> </g>
+ <g id="shape85-106" v:mID="85" v:groupContext="shape" transform="translate(478.538,12.9307) rotate(65.6291)">
+ <title>Sheet.85</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 261 L109.61 261" class="st11"/>
+ </g>
+ <g id="shape117-111" v:mID="117" v:groupContext="shape" transform="translate(247.054,-91.2818)">
+ <title>Sheet.117</title>
+ <desc>Attack Signature Length X</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="52.7082" cy="248.565" width="105.42" height="24.8704"/>
+ <rect x="0" y="236.13" width="105.416" height="24.8704" class="st7"/>
+ <text x="5.7" y="251.26" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Attack Signature Length X</text> </g>
+ </g>
+ <g id="shape122-114" v:mID="122" v:groupContext="shape" transform="translate(315.114,-164.13)">
+ <title>Sheet.122</title>
+ <desc>HTSS</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="26.943" cy="248.565" width="53.89" height="24.8704"/>
+ <rect x="0" y="236.13" width="53.8859" height="24.8704" class="st7"/>
+ <text x="14.52" y="252.16" class="st15" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>HTSS</text> </g>
+ </g>
+</svg>
diff --git a/doc/guides/prog_guide/img/member_i7.svg b/doc/guides/prog_guide/img/member_i7.svg
new file mode 100644
index 0000000..e23ae26
--- /dev/null
+++ b/doc/guides/prog_guide/img/member_i7.svg
@@ -0,0 +1,399 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.0//EN" "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd">
+<!-- Generated by Microsoft Visio 11.0, SVG Export, v1.0 memship_i7.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+ xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="8.5in" height="4.5in" viewBox="0 0 612 324"
+ xml:space="preserve" color-interpolation-filters="sRGB" class="st23">
+ <v:documentProperties v:langID="1033" v:viewMarkup="false"/>
+
+ <style type="text/css">
+ <![CDATA[
+ .st1 {visibility:visible}
+ .st2 {fill:#5b9bd5;fill-opacity:0.22;stroke:#5b9bd5;stroke-opacity:0.22;stroke-width:0.25}
+ .st3 {fill:#5b9bd5;stroke:#c8c8c8;stroke-width:0.25}
+ .st4 {fill:#feffff;font-family:Calibri;font-size:0.833336em;font-weight:bold}
+ .st5 {font-size:1em}
+ .st6 {fill:#70ad47;fill-opacity:0.5;stroke:#00b050;stroke-width:1.5}
+ .st7 {fill:none;stroke:none;stroke-width:0.25}
+ .st8 {fill:#00b050;font-family:Calibri;font-size:1.16666em}
+ .st9 {fill:none;stroke:#00b050;stroke-width:2.25}
+ .st10 {fill:#5b9bd5;font-family:Calibri;font-size:0.833336em}
+ .st11 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em}
+ .st12 {fill:#a8d08d;stroke:#c8c8c8;stroke-width:0.25}
+ .st13 {marker-end:url(#mrkr5-83);stroke:#ff0000;stroke-linecap:round;stroke-linejoin:round;stroke-width:2.25}
+ .st14 {fill:#ff0000;fill-opacity:1;stroke:#ff0000;stroke-opacity:1;stroke-width:0.47169811320755}
+ .st15 {marker-end:url(#mrkr5-95);stroke:#92d050;stroke-linecap:round;stroke-linejoin:round;stroke-width:2.25}
+ .st16 {fill:#92d050;fill-opacity:1;stroke:#92d050;stroke-opacity:1;stroke-width:0.47169811320755}
+ .st17 {fill:#00b050;font-family:Calibri;font-size:1.00001em;font-weight:bold}
+ .st18 {fill:#5b9bd5;font-family:Calibri;font-size:1.00001em}
+ .st19 {fill:none;stroke:#ff0000;stroke-width:2.25}
+ .st20 {fill:#ff0000;font-family:Calibri;font-size:1.00001em;font-weight:bold}
+ .st21 {marker-end:url(#mrkr5-123);stroke:#ff0000;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st22 {fill:#ff0000;fill-opacity:1;stroke:#ff0000;stroke-opacity:1;stroke-width:0.28409090909091}
+ .st23 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+ ]]>
+ </style>
+
+ <defs id="Markers">
+ <g id="lend5">
+ <path d="M 2 1 L 0 0 L 1.98117 -0.993387 C 1.67173 -0.364515 1.67301 0.372641 1.98465 1.00043 " style="stroke:none"/>
+ </g>
+ <marker id="mrkr5-83" class="st14" v:arrowType="5" v:arrowSize="2" v:setback="3.71" refX="-3.71" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-2.12,-2.12) "/>
+ </marker>
+ <marker id="mrkr5-95" class="st16" v:arrowType="5" v:arrowSize="2" v:setback="3.71" refX="-3.71" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-2.12,-2.12) "/>
+ </marker>
+ <marker id="mrkr5-123" class="st22" v:arrowType="5" v:arrowSize="2" v:setback="5.8" refX="-5.8" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-3.52,-3.52) "/>
+ </marker>
+ </defs>
+ <g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+ <title>Page-1</title>
+ <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+ <g id="group121-1" transform="translate(21.0294,-32.2733)" v:mID="121" v:groupContext="group">
+ <title>Sheet.121</title>
+ <g id="shape49-2" v:mID="49" v:groupContext="shape" transform="translate(460.471,-62.2267)">
+ <title>Rectangle.2</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow49-3" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="279" width="108" height="45" class="st2"/>
+ </g>
+ <rect x="0" y="279" width="108" height="45" class="st3"/>
+ </g>
+ <g id="shape50-6" v:mID="50" v:groupContext="shape" transform="translate(320.452,-18.123)">
+ <title>Rectangle.4</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow50-7" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="234" width="108" height="90" class="st2"/>
+ </g>
+ <rect x="0" y="234" width="108" height="90" class="st3"/>
+ </g>
+ <g id="shape52-10" v:mID="52" v:groupContext="shape" transform="translate(0,-31.5)">
+ <title>Rectangle.10</title>
+ <desc>Flow Keys Matching Mask 1</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="285.75" width="108" height="76.5"/>
+ <g id="shadow52-11" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="247.5" width="108" height="76.5" class="st2"/>
+ </g>
+ <rect x="0" y="247.5" width="108" height="76.5" class="st3"/>
+ <text x="12.56" y="282.75" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow Keys Matching <tspan
+ x="39.1" dy="1.2em" class="st5">Mask </tspan>1</text> </g>
+ <g id="shape53-16" v:mID="53" v:groupContext="shape" transform="translate(311.452,-27.123)">
+ <title>Sheet.53</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <rect x="0" y="310.5" width="126" height="13.5" class="st6"/>
+ </g>
+ <g id="shape54-18" v:mID="54" v:groupContext="shape" transform="translate(424.471,-26.2267)">
+ <title>Sheet.54</title>
+ <desc>Match</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="36" cy="317.25" width="72" height="13.5"/>
+ <rect x="0" y="310.5" width="72" height="13.5" class="st7"/>
+ <text x="17.68" y="321.45" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Match</text> </g>
+ <g id="shape55-21" v:mID="55" v:groupContext="shape" transform="translate(261,-247.163)">
+ <title>Sheet.55</title>
+ <desc>Flow ID1</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="27.1728" cy="315" width="54.35" height="18"/>
+ <rect x="0" y="306" width="54.3456" height="18" class="st9"/>
+ <text x="9.52" y="318" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow ID1</text> </g>
+ <g id="shape96-24" v:mID="96" v:groupContext="shape" transform="translate(0,-109.783)">
+ <title>Sheet.96</title>
+ <desc>Flow Mask 1</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="319.5" width="108" height="9"/>
+ <rect x="0" y="315" width="108" height="9" class="st7"/>
+ <text x="18.51" y="323.7" class="st11" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow Mask 1</text> </g>
+ <g id="group114-27" transform="translate(247.5,-163.783)" v:mID="114" v:groupContext="group">
+ <title>Sheet.114</title>
+ <g id="group106-28" transform="translate(0,-27)" v:mID="106" v:groupContext="group">
+ <title>Sheet.106</title>
+ <g id="shape100-29" v:mID="100" v:groupContext="shape" transform="translate(0,-13.5)">
+ <title>Rectangle.100</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow100-30" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape101-33" v:mID="101" v:groupContext="shape" transform="translate(27,-13.5)">
+ <title>Rectangle.101</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow101-34" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape102-37" v:mID="102" v:groupContext="shape" transform="translate(54,-13.5)">
+ <title>Rectangle.102</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow102-38" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape103-41" v:mID="103" v:groupContext="shape">
+ <title>Rectangle.103</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow103-42" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape104-45" v:mID="104" v:groupContext="shape" transform="translate(27,0)">
+ <title>Rectangle.104</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow104-46" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape105-49" v:mID="105" v:groupContext="shape" transform="translate(54,0)">
+ <title>Rectangle.105</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow105-50" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ </g>
+ <g id="group107-53" v:mID="107" v:groupContext="group">
+ <title>Sheet.107</title>
+ <g id="shape108-54" v:mID="108" v:groupContext="shape" transform="translate(0,-13.5)">
+ <title>Rectangle.100</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow108-55" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape109-58" v:mID="109" v:groupContext="shape" transform="translate(27,-13.5)">
+ <title>Rectangle.101</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow109-59" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape110-62" v:mID="110" v:groupContext="shape" transform="translate(54,-13.5)">
+ <title>Rectangle.102</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <g id="shadow110-63" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st12"/>
+ </g>
+ <g id="shape111-66" v:mID="111" v:groupContext="shape">
+ <title>Rectangle.103</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <g id="shadow111-67" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape112-70" v:mID="112" v:groupContext="shape" transform="translate(27,0)">
+ <title>Rectangle.104</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow112-71" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape113-74" v:mID="113" v:groupContext="shape" transform="translate(54,0)">
+ <title>Rectangle.105</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <g id="shadow113-75" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ </g>
+ </g>
+ <g id="shape89-78" v:mID="89" v:groupContext="shape" transform="translate(413.723,393.802) rotate(146.31)">
+ <title>Sheet.89</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 324 L153.9 324" class="st13"/>
+ </g>
+ <g id="shape115-84" v:mID="115" v:groupContext="shape" transform="translate(126,0)">
+ <title>Rectangle.115</title>
+ <desc>Flow Keys Matching Mask 2</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="270" width="108" height="108"/>
+ <g id="shadow115-85" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="216" width="108" height="108" class="st2"/>
+ </g>
+ <rect x="0" y="216" width="108" height="108" class="st3"/>
+ <text x="12.56" y="267" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow Keys Matching <tspan
+ x="39.1" dy="1.2em" class="st5">Mask </tspan>2</text> </g>
+ <g id="shape85-90" v:mID="85" v:groupContext="shape" transform="translate(635.321,91.2793) rotate(81.3573)">
+ <title>Sheet.85</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 324 L143.93 324" class="st15"/>
+ </g>
+ <g id="shape56-96" v:mID="56" v:groupContext="shape" transform="translate(579.175,-64.556) rotate(64.1257)">
+ <title>Sheet.56</title>
+ <path d="M0 324 L54.31 324" class="st15"/>
+ </g>
+ </g>
+ <g id="shape122-101" v:mID="122" v:groupContext="shape" transform="translate(351,-213.444)">
+ <title>Sheet.122</title>
+ <desc>HTSS with False Negative (Cache)</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="45" cy="304.722" width="90" height="38.556"/>
+ <rect x="0" y="285.444" width="90" height="38.556" class="st7"/>
+ <text x="13.29" y="301.72" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>HTSS with False <tspan
+ x="10.52" dy="1.2em" class="st5">Negative </tspan>(Cache)</text> </g>
+ <g id="shape123-105" v:mID="123" v:groupContext="shape" transform="translate(287.654,-290.556)">
+ <title>Sheet.123</title>
+ <desc>Active</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="24.1875" cy="310.5" width="48.38" height="27"/>
+ <rect x="0" y="297" width="48.375" height="27" class="st7"/>
+ <text x="8.63" y="314.1" class="st17" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Active</text> </g>
+ <g id="shape124-108" v:mID="124" v:groupContext="shape" transform="translate(278.827,-153)">
+ <title>Sheet.124</title>
+ <desc>Target for Flow ID 1</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="36.0864" cy="310.5" width="72.18" height="27"/>
+ <rect x="0" y="297" width="72.1728" height="27" class="st9"/>
+ <text x="11.93" y="306.9" class="st18" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Target for <tspan
+ x="13.54" dy="1.2em" class="st5">Flow ID </tspan>1</text> </g>
+ <g id="shape125-112" v:mID="125" v:groupContext="shape" transform="translate(155.857,-254.556)">
+ <title>Sheet.125</title>
+ <desc>Flow ID2</desc>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="27.1728" cy="315" width="54.35" height="18"/>
+ <rect x="0" y="306" width="54.3456" height="18" class="st19"/>
+ <text x="9.52" y="318" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow ID2</text> </g>
+ <g id="shape126-115" v:mID="126" v:groupContext="shape" transform="translate(153,-270)">
+ <title>Sheet.126</title>
+ <desc>New/Inactive</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="40.5" cy="310.5" width="81" height="27"/>
+ <rect x="0" y="297" width="81" height="27" class="st7"/>
+ <text x="6.77" y="314.1" class="st20" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>New/Inactive</text> </g>
+ <g id="shape127-118" v:mID="127" v:groupContext="shape" transform="translate(251.739,-239.709) rotate(14.0795)">
+ <title>Sheet.127</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 318.73 A39.2404 18 -180 0 0 49.73 320.91 L50.07 320.78" class="st21"/>
+ </g>
+ <g id="shape128-124" v:mID="128" v:groupContext="shape" transform="translate(219.24,-229.5)">
+ <title>Sheet.128</title>
+ <desc>Miss</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="20.88" cy="310.5" width="41.76" height="27"/>
+ <rect x="0" y="297" width="41.76" height="27" class="st7"/>
+ <text x="7.81" y="314.7" class="st11" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Miss</text> </g>
+ <g id="shape129-127" v:mID="129" v:groupContext="shape" transform="translate(147.029,-142.056)">
+ <title>Sheet.129</title>
+ <desc>Flow Mask 2</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="319.5" width="108" height="9"/>
+ <rect x="0" y="315" width="108" height="9" class="st7"/>
+ <text x="18.51" y="323.7" class="st11" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow Mask 2</text> </g>
+ <g id="shape130-130" v:mID="130" v:groupContext="shape" transform="translate(166.845,-18.5004) rotate(18.2325)">
+ <title>Sheet.130</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 293.46 A71.1913 104.269 -180 0 0 97.04 298.43 L97.25 298.14" class="st21"/>
+ </g>
+ <g id="shape131-135" v:mID="131" v:groupContext="shape" transform="translate(184.406,-3.04505) rotate(-3.24734)">
+ <title>Sheet.131</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 293.46 A112.345 104.269 -180 0 0 154.25 297.52 L154.52 297.28" class="st21"/>
+ </g>
+ <g id="shape132-140" v:mID="132" v:groupContext="shape" transform="translate(301.368,16.888) rotate(-25.868)">
+ <title>Sheet.132</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 293.46 A83.375 104.269 -180 0 0 113.91 298.14 L114.14 297.87" class="st21"/>
+ </g>
+ <g id="shape133-145" v:mID="133" v:groupContext="shape" transform="translate(345.029,-142.056)">
+ <title>Sheet.133</title>
+ <desc>Flow Mask X</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="319.5" width="108" height="9"/>
+ <rect x="0" y="315" width="108" height="9" class="st7"/>
+ <text x="18.43" y="323.7" class="st11" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow Mask X</text> </g>
+ <g id="shape134-148" v:mID="134" v:groupContext="shape" transform="translate(481.5,-139.5)">
+ <title>Sheet.134</title>
+ <desc>Flow Mask L</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="319.5" width="108" height="9"/>
+ <rect x="0" y="315" width="108" height="9" class="st7"/>
+ <text x="19.12" y="323.7" class="st11" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow Mask L</text> </g>
+ </g>
+</svg>
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index 40f04a1..7ff5144 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -50,6 +50,7 @@ Programmer's Guide
timer_lib
hash_lib
efd_lib
+ member_lib
lpm_lib
lpm6_lib
packet_distrib_lib
@@ -191,6 +192,19 @@ Programmer's Guide

:numref:`figure_efd11` :ref:`figure_efd11`

+:numref:`figure_membership1` :ref:`figure_membership1`
+
+:numref:`figure_membership2` :ref:`figure_membership2`
+
+:numref:`figure_membership3` :ref:`figure_membership3`
+
+:numref:`figure_membership4` :ref:`figure_membership4`
+
+:numref:`figure_membership5` :ref:`figure_membership5`
+
+:numref:`figure_membership6` :ref:`figure_membership6`
+
+:numref:`figure_membership7` :ref:`figure_membership7`

**Tables**

diff --git a/doc/guides/prog_guide/member_lib.rst b/doc/guides/prog_guide/member_lib.rst
new file mode 100644
index 0000000..48946d6
--- /dev/null
+++ b/doc/guides/prog_guide/member_lib.rst
@@ -0,0 +1,432 @@
+.. BSD LICENSE
+ Copyright(c) 2017 Intel Corporation. All rights reserved.
+ All rights reserved.
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions
+ are met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in
+ the documentation and/or other materials provided with the
+ distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ contributors may be used to endorse or promote products derived
+ from this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+.. _Member_Library:
+
+Membership Library
+=======================
+
+Introduction
+------------
+The DPDK Membership Library provides an API for DPDK applications to insert a
+new member, delete an existing member, or query the existence of a member in a
+given set, or a group of sets. For the case of a group of sets the library
+will return not only whether the element has been inserted before in one of
+the sets but also which set it belongs to. The Membership Library is an
+extension and generalization of a traditional filter structure (for example
+Bloom Filter [Member-bloom]) that has multiple usages in a wide variety of
+workloads and applications. In general, the Membership Library is a data
+structure that provides a “set-summary” on whether a member belongs to a set,
+and as discussed in details later, there are two advantages of using such a
+set-summary rather than operating on a “full-blown” complete list of elements:
+first, it has a much smaller storage requirement than storing the whole list of
+elements themselves, and secondly checking an element membership (or other
+operations) in this set-summary is much faster than checking it for the
+original full-blown complete list of elements.
+
+We use the term “Set-Summary” in this guide to refer to the space-efficient,
+probabilistic membership data structure that is provided by the library. A
+membership test for an element will return the set this element belongs to or
+null (meaning not found) with very high probability of accuracy. Set-summary
+is a fundamental data aggregation component that can be used in many network
+(and other) applications. It is a crucial structure to address performance and
+scalability issues of diverse network applications including overlay networks,
+data-centric networks, flow table summaries, network statistics and
+traffic monitoring. A set-summary is useful for applications who need to
+include a list of elements while a complete list requires too much space
+and/or too much processing cost. In these situations, the set-summary works as
+a lossy hash-based representation of a set of members. It can dramatically
+reduce space requirement and significantly improve the performance of set
+membership queries at the cost of introducing a very small membership test error
+probability.
+
+.. _figure_membership1:
+.. figure:: img/member_i1.*
+
+ Example Usages of Membership Library
+
+We believe that there are various usages for a Membership Library in a very
+large set of applications and workloads. Interested readers can refer to
+[Member-survey] for a survey of possible networking usages. The above figure
+provide a small set of examples of using the Membership Library. Sub-figure(a)
+depicts a distributed web cache architecture where a collection of proxies
+attempt to share their web caches (cached from a set of backend web servers) to
+provide faster responses to clients, and the proxies uses the Membership
+Library to share summaries of what web pages they are caching. With the
+Membership Library, a proxy receiving an \http request will inquire the
+set-summary to find its location and quickly determine whether to retrieve the
+requested web page from a nearby proxy or from a backend web server.
+Sub-figure(b) depicts another example for using the Membership Library to
+prevent routing loops which is typically done using slow TTL countdown and
+dropping packets when TTL expires. As shown in Sub-figure(b), an embedded
+set-summary in the packet header itself can be used to summarize the set of
+nodes a packet has gone through, and each node upon receiving a packet can check
+whether its id is a member of the set of visited nodes, and if it is then a
+routing loop is detected. Sub-Figure(c) presents another usage of Membership
+Library to load balance flows to worker threads with in-order guarantee where a
+set-summary is used to query if a packet belongs to an existing flow or a new
+flow. Packets belonging to a new flow are forwarded to the current least loaded
+worker thread, while those belonging to an existing flow are forwarded to the
+pre-assigned thread to guarantee in-order processing. Sub-figure(d) highlights
+yet another usage example in the database domain where a set-summary is used to
+determine joins between sets instead of creating a join by comparing each
+element of a set against the other elements in a different set, a join is done
+on the summaries since they can efficiently encode members of a given set.
+
+We are including a configurable Membership Library in DPDK to cover set
+membership functionality for both a single set and multi-set scenarios. The
+library is optimized to support the customer network applications which require
+membership checking functionality. In this guide, we will cover two set-summary
+schemes including vector of Bloom Filters and Hash-Table based
+set-summary schemes with and without false negative probability, followed by
+a brief discussion of the Membership Library API.
+
+Vector of Bloom Filters
+--------------------------
+
+Bloom Filter (BF) [Member-bloom] is a well-known space-efficient
+probabilistic data structure that answers set membership queries (test whether
+an element is a member of a set) with some probability of false positives and
+zero false negatives; a query for an element returns either it is "possibly in
+a set" (with very high probability) or "definitely not in a set".
+
+The BF is a method for representing a set of n elements (for example flow keys
+in network applications domain) to support membership queries. The idea of BF is
+to allocate a bit-vector v with m bits, which are initially all set to 0. Then
+it chooses k independent hash functions h1, h2, … hk with hash values range from
+1 to m to perform hashing calculations on each element. Every time when an
+element X being inserted into the set, the bits at positions h1(X), h2(X), …
+hk(X) in v are set to 1 (any particular bit might be set to 1 multiple times
+for multiple different inserted elements). Given a query for any element Y, the
+bits at positions h1(Y), h2(Y), ... hk(Y) are checked. If any of them is 0,
+then Y is definitely not in the set. Otherwise there is a high probability that
+Y is a member of the set with certain false positive probability. As shown in
+the next equation, the false positive probability can be made arbitrarily small
+by changing the number of hash functions (k) and the vector length (m).
+
+.. _figure_membership2:
+.. figure:: img/member_i2.*
+
+ Bloom Filter False Positive Probability
+
+Without BF, an accurate membership testing could involve a costly hash table
+lookup and full element comparison. The advantage of using a BF is to simplify
+the membership test into a series of hash calculations and memory accesses for a
+small bit-vector, which can be easily optimized. Hence the lookup throughput
+(set membership test) can be significantly faster than a normal hash table
+lookup with element comparison.
+
+.. _figure_membership3:
+.. figure:: img/member_i3.*
+
+ Detecting Routing Loops Using BF
+
+BF is used for applications that need only one set, and the
+membership of elements is checked against the BF. The example discussed
+in the above figure is one example of potential applications that uses only one
+set to capture the node IDs that have been visited so far by the packet. Each
+node will then check this embedded BF in the packet header for its own id, and
+if the BF indicates that the current node is definitely not in the set then a
+loop-free route is guaranteed.
+
+
+.. _figure_membership4:
+.. figure:: img/member_i4.*
+
+ Vector Bloom Filter (vBF) Overview
+
+To support membership test for both multiple sets and single set,
+the library implements Vector Bloom Filter (vBF) scheme.
+vBF basically composes of multiple bloom filters as a vector of bloom filers.
+The membership test is conducted on all of the
+bloom filters concurrently to determine which set(s) it belongs to or none of
+them. The basic idea of vBF is shown in the above figure where an element is
+used to address multiple bloom filters concurrently and the bloom filter
+index(es) with a hit is returned.
+
+.. _figure_membership5:
+.. figure:: img/member_i5.*
+
+ vBF for Flow Scheduling to Worker Thread
+
+As previously mentioned, there are many usages of such structure. vBF is used
+for applications that needs to check membership against multiple sets
+simultaneously. The example discussed in the above figure uses a set to capture
+all flows being assigned for processing at a given worker thread. Upon receiving
+a packet the vBF is used to quickly figure out if this packet belongs to a new flow
+so as to be forwarded to the current least loaded worker thread, or otherwise it
+should be queued for an existing thread to guarantee in-order processing (i.e.
+the property of vBF to indicate right away that a given flow is a new one or
+not is critical to minimize response time latency).
+
+It should be noted that vBF can be implemented using a set of single bloom
+filters with sequential lookup of each BF. However, being able to concurrently
+search all set-summaries is a big throughput advantage. In the library, certain
+parallelism is realized by the implementation of checking all bloom filters
+together.
+
+
+Hash-Table based Set-Summaries
+---------------------------------
+
+Hash-table based set-summary (HTSS) is another scheme in the membership library.
+Cuckoo filter [Member-cfilter] is an example of hash-table based set summary.
+HTSS can be easily extended to support multi-set membership testing like what
+vBF does. Meanwhile, HTSS can easily outperform vBF when the number of sets is
+large, since HTSS uses a single hash table for membership testing while vBF
+requires testing a series of Bloom Filters each corresponding to one set.
+
+.. _figure_membership6:
+.. figure:: img/member_i6.*
+
+ Using HTSS for Attack Signature Matching
+
+As shown in the above figure, attack signature matching where each set
+represents a certain signature length (for correctness of this example, an
+attack signature should not be a subset of another one) in the payload is a good
+example for using HTSS with 0% false negative (i.e., when an element returns not
+found, it has a 100% certainty that it is not a member of any set). The packet
+inspection application benefits from knowing right away that the current payload
+does not match any attack signatures in the database to establish its
+legitimacy, otherwise a deep inspection of the packet is needed.
+
+HTSS employs a similar but simpler data structure to a traditional hash table,
+and the major difference is that HTSS stores only the signatures but not the
+full keys/elements which can significantly reduce the footprint of the table.
+Along with the signature, HTSS also stores a value to indicate the target set.
+When looking up for an element, the element is hashed and the HTSS is addressed
+to retrieve the signature stored. If the signature matches then the value is
+retrieved corresponding to the index of the target set which the element belongs
+to. Because signatures can collide, HTSS can still has false positive
+probability similar to vBF. Furthermore, if elements are allowed to be
+overwritten or evicted when the hash table becomes full, it will also have a
+false negative probability. We discuss this case in the next section.
+
+Set-Summaries with False Negative Probability
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As previously discussed, traditional set-summaries (e.g. Bloom Filters ) do not
+have a false negative probability, i.e., it is 100% certain when an element
+returns “not to be present” for a given set. However, the Membership Library
+also supports a set-summary probabilistic data structure based on HTSS which
+allows for false negative probability.
+
+
+In HTSS, when the hash table becomes full, keys/elements will fail to be added
+into the table and the hash table has to be resized to accommodate for these new
+elements, which can be expensive. However, if we allow new elements to overwrite
+or evict existing elements (as a cache typically does), then the resulting
+set-summary will begin to have false negative probability. This is because the
+element that was evicted from the set-summary may still be present in the target
+set. For subsequent inquiries the set-summary will falsely report the element
+not being in the set, hence having a false negative probability.
+
+The major usage of HTSS with false negative is to use it as a cache for
+distributing elements to different target sets. By allowing HTSS to evict old
+elements, the set-summary can keep track of the most recent elements
+(i.e. active) as a cache typically does. Old inactive elements (infrequently
+used elements) will automatically and eventually get evicted from the
+set-summary. It worth noting that the set-summary still has false positive
+probability, which means the application either can tolerate certain false positive
+or it has fall-back path when false positive happens.
+
+.. _figure_membership7:
+.. figure:: img/member_i7.*
+
+ Using HTSS with False Negatives for Wild Card Classification
+
+HTSS with false negative (i.e. a cache) also has its wide set of applications.
+For example wild card flow classification (e.g. ACL rules) highlighted in the
+above figure is an example of such application. In that case each target set
+represents a sub-table with rules defined by a certain flow mask. The flow masks
+are non-overlapping, and for flows matching more than one rule only the highest
+priority one is inserted in the corresponding sub-table (interested readers can
+refer to the Open vSwitch (OvS) design of Mega Flow Cache (MFC) [Member-OvS]
+for further details). Typically the rules will have a large number of distinct
+unique masks and hence, a large number of target sets each corresponding to one
+mask. Because the active set of flows varies widely based on the network
+traffic, HTSS with false negative will act as a cache for <flowid, target ACL
+sub-table> pair for the current active set of flows. When a miss occurs (as
+shown in red in the above figure) the sub-tables will be searched sequentially
+one by one for a possible match, and when found the flow key and target
+sub-table will be inserted into the set-summary (i.e. cache insertion) so
+subsequent packets from the same flow don’t incur the overhead of the
+sequential search of sub-tables.
+
+Library API Overview
+--------------------
+The design goal of the Membership Library API is to be as generic as possible to
+support all the different types of set-summaries we discussed in previous
+sections and beyond. Fundamentally, the APIs need to include creation,
+insertion, deletion, and lookup.
+
+.. Set-summary Type Query
+.. ~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. *void rte\_ms\_type\_query(enum rte\_ms\_setsum\_type\*\* types)*
+
+.. This function intends to serve as a convenient tool to query which set-summary
+.. types are supported in the current version of DPDK. Since the implementations of
+.. the set-summaries can vary, initial versions of the Membership Library may only
+.. support a subset of the set-summaries we discussed previously. Programmers could
+.. use this API to decide which type is available to use in the current version of DPDK.
+
+Set-summary Create
+~~~~~~~~~~~~~~~~~~~~~
+
+*rte\_member\_create* function is used to create a set-summary structure, the input parameter
+is a struct to pass in parameters that needed to initialize the set-summary, while the function returns the
+pointer to the created set-summary or NULL if the creation failed.
+
+The general input arguments used when creating the set-summary should include *name*
+which is the name of the created set-summary, *type* which is one of the types
+supported by the library (e.g. RTE\_MEMBER\_TYPE\_HT for HTSS or RTE\_MEMBER\_TYPE\_VBF for vBF), and *key\_len*
+which is the length of the element/key. There are other parameters
+are only used for certain type of set-summary, or means slightly different for different types of set-summary.
+For example, *num\_keys* parameter means the total number of entries for Hash table based set-summary.
+However, for bloom filter, this value means the expected number of keys that could be
+inserted into the bloom filter(s). The value is used to calculate the size of each
+bloom filter.
+We also pass two seeds: *prim\_hash\_seed* and
+*sec\_hash\_seed* for the primary and secondary hash functions to calculate two independent hash values.
+*socket\_id* parameter is the NUMA socket ID for the memory used to create the
+set-summary. For HTSS, another parameter *iscache* is used to indicate
+if this set-summary is a cache (i.e. with false negative probability) or not.
+For vBF, extra parameters are needed. For example, *num\_set* is the number of
+sets needed to initialize the vector bloom filters. This number is equal to the
+number of bloom filters will be created.
+*false\_pos\_rate* is the false positive rate. num\_keys and false\_pos\_rate will be used to determine
+the number of hash functions and the bloom filter size.
+
+
+Set-summary Element Insertion
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. *rte\_membership\_add(const void *setsum, const void *key, MEMBERSHIP\_TARGET\_TYPE target\_id)*
+
+*rte\_member\_add* is the function to insert an element/key in a set-summary structure, if it fails an
+error is returned. For success the returned value is deferent based on the
+set-summary mode to provide extra information for the users. For vBF
+mode, a return value of 0 means a successful insert. For HTSS mode without false negative, the insert
+could fail with -ENOSPC if the table is full. With false negative (i.e. cache mode),
+for insert that does not cause any eviction (i.e. no overwriting happens to an
+existing entry) the return value is 0. For insertion that causes eviction, the return
+value is 1 to indicate such situation, but it is not an error.
+
+The input arguments for the function should include the *key* which is a pointer to the element/key that needs to
+be added to the set-summary, and *set\_id* which is the set id associated
+with the key that needs to be added.
+
+
+Set-summary Element Lookup
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+.. *rte\_membership\_lookup(const void \*setsum, const void \*key, MEMBERSHIP\_TARGET\_TYPE \*target\_id)*
+
+*rte\_member\_lookup* looks up a single key/element in the set-summary structure. It
+returns as soon as the first match is found. The return value is 1 if a
+match is found and 0 otherwise. The arguments for the function include *key* which is a pointer to the
+element/key that needs to be looked up, and *set\_id* which is used to return the
+first target set id where the key has matched, if any.
+
+.. *rte\_membership\_lookup\_bulk(const void \*setsum, const void \*\*keys, uint32\_t num\_keys, MEMBERSHIP\_TARGET\_TYPE \*target\_ids)*
+
+*rte\_member\_lookup\_bulk* is the function to look up a bulk of keys/elements in the
+set-summary structure for their first match. Each key lookup returns as soon as the first match is found. The
+return value is the number of keys that find a match. The arguments of the
+function include *keys* which is a pointer to a bulk of keys that are to be looked up,
+*num\_keys* is the number
+of keys that will be looked up, and *set\_ids* are the return target set
+ids for the first match found for each of the input keys. *set\_ids* is an array
+needs to be sized according to the *num\_keys*. If there is no match, the set id
+for that key will be set to RTE_MEMBER_NO_MATCH.
+
+.. *rte\_membership\_lookup\_multi(const void \*setsum, const void \*key, MEMBERSHIP\_TARGET\_TYPE \*target\_id)*
+
+*rte\_member\_lookup\_multi* function looks up a single key/element in the
+set-summary structure for multiple matches. It
+returns ALL the matches (possibly more than one) found for this key when it
+is matched against all target sets (it worth noting that for cache mode HTSS,
+the current implementation matches at most one target set). The return value is
+the number of matches
+that was found for this key (for cache mode HTSS the return value
+should be at most 1). The arguments for the function include *key* which is a pointer to the
+element/key that needs to be looked up, *match\_per\_key* which is to indicate maximum number of matches
+the user expect to find for each key, and *set\_id* which is used to return all
+target set ids where the key has matched, if any. The *set\_id* array should be sized
+according to *match\_per\_key*. For vBF, maximum number of matches per key is equal
+to the number of sets. For HTSS, maximum number of matches per key is equal to two times
+entry count per bucket. *match\_per\_key* should be equal or smaller than maximum number of
+possible matches.
+
+.. *rte\_membership\_lookup\_multi\_bulk(const void \*setsum, const void \*\*keys, uint32\_t num\_keys, uint32\_t max\_match\_per\_key, uint32\_t \*match\_count, MEMBERSHIP\_TARGET\_TYPE \*target\_ids)*
+
+*rte\_membership\_lookup\_multi\_bulk* function looks up a bulk of keys/elements elements in the
+set-summary structure for multiple matches, each key lookup returns ALL the matches (possibly more
+than one) found for this key when it is matched against all target sets (cache mode HTSS
+matches at most one target set). The
+return value is the number of keys that find one or more matches in the
+set-summary structure. The arguments of the
+function include *keys* which is
+a pointer to a bulk of keys that are to be looked up, *num\_keys* is the number
+of keys that will be looked up, *match\_per\_key* is the possible
+max number of matches for each key, *match\_count* which is the returned number
+of matches for each key, and *set\_ids* are the returned target set
+ids for all matches found for each keys. *set\_ids* is 2-D array
+that for each key, a 1-D array should be sized according to *match\_per\_key*.
+*match\_per\_key* should be equal or smaller than maximum number of
+possible matches, similar to *rte\_member\_lookup\_multi*.
+
+
+Set-summary Element Delete
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. *rte\_membership\_delete(void \*setsum, const void \*key, MEMBERSHIP\_TARGET\_TYPE target\_id)*
+
+*rte\_membership\_delete* function deletes an element/key from a set-summary structure, if it fails
+an error is returned. The input arguments should include *key* which is a pointer to the
+element/key that needs to be deleted from the set-summary, and *set\_id*
+which is the set id associated with the key to delete. It worth noting that current
+implementation of vBF does not support deletion [1]_. An error code -EINVAL will be returned.
+
+.. [1] Traditional bloom filter does not support proactive deletion. Supporting proactive deletion require additional implementation and performance overhead.
+
+References
+-----------
+
+[Member-bloom] B H Bloom, "Space/Time Trade-offs in Hash Coding with Allowable Errors," Communications of the ACM, 1970.
+
+[Member-survey] A Broder and M Mitzenmacher, "Network Applications of Bloom Filters: A Survey," in Internet Mathematics, 2005.
+
+[Member-cfilter] B Fan, D G Andersen and M Kaminsky, "Cuckoo Filter: Practically Better Than Bloom," in Conference on emerging Networking Experiments and Technologies, 2014.
+
+[Member-OvS] B Pfaff, "The Design and Implementation of Open vSwitch," in NSDI, 2015.
\ No newline at end of file
diff --git a/doc/guides/rel_notes/release_17_11.rst b/doc/guides/rel_notes/release_17_11.rst
index 170f4f9..4002df3 100644
--- a/doc/guides/rel_notes/release_17_11.rst
+++ b/doc/guides/rel_notes/release_17_11.rst
@@ -41,6 +41,23 @@ New Features
Also, make sure to start the actual text at the margin.
=========================================================

+* **Added Membership library (rte_member).**
+
+ Added membership library. It provides an API for DPDK applications to insert a
+ new member, delete an existing member, or query the existence of a member in a
+ given set, or a group of sets. For the case of a group of sets the library
+ will return not only whether the element has been inserted before in one of
+ the sets but also which set it belongs to.
+
+ The Membership Library is an extension and generalization of a traditional
+ filter (for example Bloom Filter) structure that has multiple usages in a wide
+ variety of workloads and applications. In general, the Membership Library is a
+ data structure that provides a “set-summary” and responds to set-membership
+ queries whether a certain member belongs to a set(s).
+
+ See the :ref:`Membership Library <Member_Library>` documentation in
+ the Programmers Guide document, for more information.
+

Resolved Issues
---------------
--
2.7.4
Stephen Hemminger
2017-08-22 04:01:32 UTC
Permalink
On Mon, 21 Aug 2017 17:19:46 -0700
Post by Yipeng Wang
This patch set implements two types of set-summaries, i.e., hash-table based
set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the
non-cache and cache modes. The non-cache mode can incur a small chance of
false-positives which is the case when the set-summary indicates a key belongs
to a given set while actually it is not. The cache mode can also have
false-negatives in addition to false-positives. False-negatives means the case
when the set-summary indicates a key does not belong to a given set while
actually it does. This happens because cache mode allows new key to evict
existing keys. vBF only has false-positives similar to the non-cache HTSS.
However, one can set the false-positive rate arbitrarily. HTSS's
false-positive rate is determined by the hash-table size and the signature size.
I don't think it makes sense to merge two different types of tables in one
API. Especially in DPDK where every cycle counts. You are taking an extra branch
on each lookup. The user of this API is likely to know exactly what type
of objects and look are desired.
Wang, Yipeng1
2017-08-23 02:58:51 UTC
Permalink
Hi Stephen,

Thanks for the comments. We will remove the needless return initialization in next version.

For the unified API concern, we think the current implementation has two benefits: 1. It is easier to extend the library
with new types of filters without adding a lot of top level APIs every time. 2. When users switch between different
types in their application, they do not need to switch between function calls. They can reuse same code for
different types.

However, we agree with you that a switch case in the library affects the performance. We
did a quick test and here is the results (cycles/operation):
lookup lookup_bulk(16 batch) lookup_multi lookup_multi_bulk
switch_case 50 36 57 45
direct call 47 35 54 45

There will be 3 cycle difference for non-bulk version lookup. I guess if users usually use bulk version, it maybe not
a big concern, but single key lookup indeed slower. If you think the benefit we mentioned does not outweigh the
performance slowdown, we would like to make the change.

We also considered using function pointers to get rid of switch case, but it will be unfriendly to the multi-process usages.

Please comments.

Thanks
Yipeng
-----Original Message-----
Sent: Monday, August 21, 2017 9:02 PM
Subject: Re: [PATCH 0/7] Add Membership Library
On Mon, 21 Aug 2017 17:19:46 -0700
Post by Yipeng Wang
This patch set implements two types of set-summaries, i.e., hash-table
based set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports
both the non-cache and cache modes. The non-cache mode can incur a
small chance of false-positives which is the case when the set-summary
indicates a key belongs to a given set while actually it is not. The
cache mode can also have false-negatives in addition to
false-positives. False-negatives means the case when the set-summary
indicates a key does not belong to a given set while actually it does.
This happens because cache mode allows new key to evict existing keys. vBF
only has false-positives similar to the non-cache HTSS.
Post by Yipeng Wang
However, one can set the false-positive rate arbitrarily. HTSS's
false-positive rate is determined by the hash-table size and the signature size.
I don't think it makes sense to merge two different types of tables in one API.
Especially in DPDK where every cycle counts. You are taking an extra branch on
each lookup. The user of this API is likely to know exactly what type of objects
and look are desired.
Yipeng Wang
2017-09-02 01:24:34 UTC
Permalink
DPDK Membership Library provides an API that can be used by many DPDK
applications to conduct one or more set-membership tests (we mention some
possible use cases below, but interested readers can refer to
[1] for a wider survey of use cases).

The basic functionalities of the Membership Library include
inserting a new member, deleting an existing member, and querying the existence
of a member. The query result would indicate with high accuracy which specific
set this member belongs to among a group of sets.

The Membership Library is an extension and generalization of traditional filter
data structures [2,3], which maintain a space-efficient “set-summary”.
There are two advantages of using such a set-summary rather than operating on a
“full-blown” complete list of elements: firstly it has a much smaller storage
requirement than storing the whole list of elements, and secondly set membership
tests (or other operations) is much more efficient than searching through the
complete list of elements.

A membership test for an element will return the set this element belongs to if
found (or return "not-found") with high accuracy. If needed, the accuracy of the
membership tests could be further increased with larger storage space.
Set-summary is a fundamental data aggregation component that can be used in many
network applications. It is a crucial structure to address performance and
scalability issues of diverse applications including overlay networks, wild card
flow classification, web-caches, load balancing, connection tracking,
data-centric networks, flow table summaries, network statistics and traffic
monitoring. Our Proof of Concept (PoC) using set-summary to optimize flow lookup
in Open vSwitch (OvS) shows a speedup of about 2-3X.

This patch set implements two types of set-summaries, i.e., hash-table based
set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the
non-cache and cache modes. The non-cache mode can incur a small chance of
false-positives which is the case when the set-summary indicates a key belongs
to a given set while actually it is not. The cache mode can also have
false-negatives in addition to false-positives. False-negatives means the case
when the set-summary indicates a key does not belong to a given set while
actually it does. This happens because cache mode allows new key to evict
existing keys. vBF only has false-positives similar to the non-cache HTSS.
However, one can set the false-positive rate arbitrarily. HTSS's
false-positive rate is determined by the hash-table size and the signature size.

[1] A Broder and M Mitzenmacher, “Network Applications of Bloom Filters: A
Survey,” in Internet Mathematics, 2005.

[2] B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,”
Communications of the ACM, 1970.

[3] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than
Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.


v2:
- test: add bad parameter test functions to test the creation fail case.
- test: add find existing test.
- member lib: Add num_entries check in rte_member_create_ht
- member lib: Add multiple checks for rte_member_create_vbf to avoid divide-by-0
- member lib: remove unnecessary ret from rte_member.c according to Stephen's
comment.
- member lib: change the vBF creation function to be not too conservative.
Previous algorithm is too conservative on false positive.
- member lib: fix uninitilize issue fail gcc-4.8 in rte_member_find_existing.
- member lib: add rte_member_find_existing to version map.
- makefile: lib/librte_member/Makefile: change include order according to
Luca's comment
- documentation: update the programmer guide for membership library

Yipeng Wang (7):
member: implement main API
member: implement HT mode
member: implement vBF mode
member: add AVX for HT mode
member: enable the library
test/member: add functional and perf tests
doc: add membership documentation

MAINTAINERS | 7 +
config/common_base | 5 +
doc/api/doxy-api-index.md | 3 +-
doc/api/doxy-api.conf | 1 +
doc/guides/prog_guide/img/member_i1.svg | 1613 ++++++++++++++++++++++++++++++
doc/guides/prog_guide/img/member_i2.svg | 36 +
doc/guides/prog_guide/img/member_i3.svg | 148 +++
doc/guides/prog_guide/img/member_i4.svg | 450 +++++++++
doc/guides/prog_guide/img/member_i5.svg | 163 +++
doc/guides/prog_guide/img/member_i6.svg | 332 ++++++
doc/guides/prog_guide/img/member_i7.svg | 399 ++++++++
doc/guides/prog_guide/index.rst | 14 +
doc/guides/prog_guide/member_lib.rst | 440 ++++++++
doc/guides/rel_notes/release_17_11.rst | 17 +
lib/Makefile | 2 +
lib/librte_eal/common/eal_common_log.c | 1 +
lib/librte_eal/common/include/rte_log.h | 1 +
lib/librte_member/Makefile | 51 +
lib/librte_member/rte_member.c | 342 +++++++
lib/librte_member/rte_member.h | 518 ++++++++++
lib/librte_member/rte_member_ht.c | 569 +++++++++++
lib/librte_member/rte_member_ht.h | 115 +++
lib/librte_member/rte_member_vbf.c | 350 +++++++
lib/librte_member/rte_member_vbf.h | 85 ++
lib/librte_member/rte_member_version.map | 16 +
lib/librte_member/rte_member_x86.h | 111 ++
mk/rte.app.mk | 2 +
test/test/Makefile | 3 +
test/test/test_member.c | 682 +++++++++++++
test/test/test_member_perf.c | 643 ++++++++++++
30 files changed, 7118 insertions(+), 1 deletion(-)
create mode 100644 doc/guides/prog_guide/img/member_i1.svg
create mode 100644 doc/guides/prog_guide/img/member_i2.svg
create mode 100644 doc/guides/prog_guide/img/member_i3.svg
create mode 100644 doc/guides/prog_guide/img/member_i4.svg
create mode 100644 doc/guides/prog_guide/img/member_i5.svg
create mode 100644 doc/guides/prog_guide/img/member_i6.svg
create mode 100644 doc/guides/prog_guide/img/member_i7.svg
create mode 100644 doc/guides/prog_guide/member_lib.rst
create mode 100644 lib/librte_member/Makefile
create mode 100644 lib/librte_member/rte_member.c
create mode 100644 lib/librte_member/rte_member.h
create mode 100644 lib/librte_member/rte_member_ht.c
create mode 100644 lib/librte_member/rte_member_ht.h
create mode 100644 lib/librte_member/rte_member_vbf.c
create mode 100644 lib/librte_member/rte_member_vbf.h
create mode 100644 lib/librte_member/rte_member_version.map
create mode 100644 lib/librte_member/rte_member_x86.h
create mode 100644 test/test/test_member.c
create mode 100644 test/test/test_member_perf.c
--
2.7.4
Yipeng Wang
2017-09-02 01:24:35 UTC
Permalink
Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a
set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the
set-summary.

The results of the membership test is not 100% accurate. Certain
false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set-summary"
is memory efficient and fast on lookup.

This patch add the main API definition.

Signed-off-by: Yipeng Wang <***@intel.com>
---
lib/Makefile | 2 +
lib/librte_eal/common/eal_common_log.c | 1 +
lib/librte_eal/common/include/rte_log.h | 1 +
lib/librte_member/Makefile | 49 +++
lib/librte_member/rte_member.c | 342 ++++++++++++++++++++
lib/librte_member/rte_member.h | 518 +++++++++++++++++++++++++++++++
lib/librte_member/rte_member_version.map | 16 +
7 files changed, 929 insertions(+)
create mode 100644 lib/librte_member/Makefile
create mode 100644 lib/librte_member/rte_member.c
create mode 100644 lib/librte_member/rte_member.h
create mode 100644 lib/librte_member/rte_member_version.map

diff --git a/lib/Makefile b/lib/Makefile
index 86caba1..c82033a 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -108,6 +108,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += librte_reorder
DEPDIRS-librte_reorder := librte_eal librte_mempool librte_mbuf
DIRS-$(CONFIG_RTE_LIBRTE_PDUMP) += librte_pdump
DEPDIRS-librte_pdump := librte_eal librte_mempool librte_mbuf librte_ether
+DIRS-$(CONFIG_RTE_LIBRTE_MEMBER) += librte_member
+DEPDIRS-librte_member := librte_eal librte_hash

ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y)
DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
diff --git a/lib/librte_eal/common/eal_common_log.c b/lib/librte_eal/common/eal_common_log.c
index 0e3b932..90db6a3 100644
--- a/lib/librte_eal/common/eal_common_log.c
+++ b/lib/librte_eal/common/eal_common_log.c
@@ -279,6 +279,7 @@ static const struct logtype logtype_strings[] = {
{RTE_LOGTYPE_CRYPTODEV, "cryptodev"},
{RTE_LOGTYPE_EFD, "efd"},
{RTE_LOGTYPE_EVENTDEV, "eventdev"},
+ {RTE_LOGTYPE_MEMBER, "member"},
{RTE_LOGTYPE_USER1, "user1"},
{RTE_LOGTYPE_USER2, "user2"},
{RTE_LOGTYPE_USER3, "user3"},
diff --git a/lib/librte_eal/common/include/rte_log.h b/lib/librte_eal/common/include/rte_log.h
index ec8dba7..ce1a3d0 100644
--- a/lib/librte_eal/common/include/rte_log.h
+++ b/lib/librte_eal/common/include/rte_log.h
@@ -87,6 +87,7 @@ extern struct rte_logs rte_logs;
#define RTE_LOGTYPE_CRYPTODEV 17 /**< Log related to cryptodev. */
#define RTE_LOGTYPE_EFD 18 /**< Log related to EFD. */
#define RTE_LOGTYPE_EVENTDEV 19 /**< Log related to eventdev. */
+#define RTE_LOGTYPE_MEMBER 20 /**< Log related to membership. */

/* these log types can be used in an application */
#define RTE_LOGTYPE_USER1 24 /**< User-defined log type 1. */
diff --git a/lib/librte_member/Makefile b/lib/librte_member/Makefile
new file mode 100644
index 0000000..1a79eaa
--- /dev/null
+++ b/lib/librte_member/Makefile
@@ -0,0 +1,49 @@
+# BSD LICENSE
+#
+# Copyright(c) 2017 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in
+# the documentation and/or other materials provided with the
+# distribution.
+# * Neither the name of Intel Corporation nor the names of its
+# contributors may be used to endorse or promote products derived
+# from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS := -I$(SRCDIR) $(CFLAGS)
+CFLAGS += $(WERROR_FLAGS) -O3
+
+EXPORT_MAP := rte_member_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_member/rte_member.c b/lib/librte_member/rte_member.c
new file mode 100644
index 0000000..71b066d
--- /dev/null
+++ b/lib/librte_member/rte_member.c
@@ -0,0 +1,342 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <string.h>
+
+#include <rte_eal.h>
+#include <rte_eal_memconfig.h>
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_malloc.h>
+#include <rte_errno.h>
+
+#include "rte_member.h"
+#include "rte_member_ht.h"
+#include "rte_member_vbf.h"
+
+TAILQ_HEAD(rte_member_list, rte_tailq_entry);
+static struct rte_tailq_elem rte_member_tailq = {
+ .name = "RTE_MEMBER",
+};
+EAL_REGISTER_TAILQ(rte_member_tailq)
+
+
+void *
+rte_member_find_existing(const char *name)
+{
+ struct rte_member_setsum *setsum = NULL;
+ struct rte_tailq_entry *te;
+ struct rte_member_list *member_list;
+
+ member_list = RTE_TAILQ_CAST(rte_member_tailq.head, rte_member_list);
+
+ rte_rwlock_read_lock(RTE_EAL_TAILQ_RWLOCK);
+ TAILQ_FOREACH(te, member_list, next) {
+ setsum = (struct rte_member_setsum *) te->data;
+ if (strncmp(name, setsum->name, RTE_MEMBER_NAMESIZE) == 0)
+ break;
+ }
+ rte_rwlock_read_unlock(RTE_EAL_TAILQ_RWLOCK);
+
+ if (te == NULL) {
+ rte_errno = ENOENT;
+ return NULL;
+ }
+ return setsum;
+}
+
+void
+rte_member_free(void *ss)
+{
+ struct rte_member_setsum *setsum;
+ struct rte_member_list *member_list = NULL;
+ struct rte_tailq_entry *te;
+
+ if (ss == NULL)
+ return;
+ setsum = ss;
+ member_list = RTE_TAILQ_CAST(rte_member_tailq.head, rte_member_list);
+ rte_rwlock_write_lock(RTE_EAL_TAILQ_RWLOCK);
+ TAILQ_FOREACH(te, member_list, next) {
+ if (te->data == ss)
+ break;
+ }
+ if (te == NULL) {
+ rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+ return;
+ }
+ TAILQ_REMOVE(member_list, te, next);
+ rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ rte_member_free_ht(setsum);
+ break;
+ case RTE_MEMBER_TYPE_VBF:
+ rte_member_free_vbf(setsum);
+ break;
+ default:
+ break;
+ }
+ rte_free(setsum);
+ rte_free(te);
+}
+
+
+void *
+rte_member_create(const struct rte_member_parameters *params)
+{
+ struct rte_tailq_entry *te;
+ struct rte_member_list *member_list = NULL;
+ struct rte_member_setsum *setsum = NULL;
+ int ret;
+
+ if (params == NULL) {
+ rte_errno = EINVAL;
+ return NULL;
+ }
+
+ if ((params->key_len == 0)) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, MEMBER,
+ "Memship create with invalid parameters\n");
+ return NULL;
+ }
+
+ member_list = RTE_TAILQ_CAST(rte_member_tailq.head, rte_member_list);
+
+ rte_rwlock_write_lock(RTE_EAL_TAILQ_RWLOCK);
+
+ TAILQ_FOREACH(te, member_list, next) {
+ setsum = (struct rte_member_setsum *) te->data;
+ if (strncmp(params->name, setsum->name,
+ RTE_MEMBER_NAMESIZE) == 0)
+ break;
+ }
+ setsum = NULL;
+ if (te != NULL) {
+ rte_errno = EEXIST;
+ te = NULL;
+ goto error_unlock_exit;
+ }
+ te = rte_zmalloc("MEMBER_TAILQ_ENTRY", sizeof(*te), 0);
+ if (te == NULL) {
+ RTE_LOG(ERR, MEMBER, "tailq entry allocation failed\n");
+ goto error_unlock_exit;
+ }
+
+ /* Create a new setsum structure */
+ setsum = (struct rte_member_setsum *) rte_zmalloc_socket(params->name,
+ sizeof(struct rte_member_setsum), RTE_CACHE_LINE_SIZE,
+ params->socket_id);
+ if (setsum == NULL) {
+ RTE_LOG(ERR, MEMBER, "Create setsummary failed\n");
+ goto error_unlock_exit;
+ }
+ setsum->type = params->type;
+ setsum->socket_id = params->socket_id;
+ setsum->key_len = params->key_len;
+ setsum->num_set = params->num_set;
+ setsum->name = params->name;
+ setsum->prim_hash_seed = params->prim_hash_seed;
+ setsum->sec_hash_seed = params->sec_hash_seed;
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ ret = rte_member_create_ht(setsum, params);
+ break;
+ case RTE_MEMBER_TYPE_VBF:
+ ret = rte_member_create_vbf(setsum, params);
+ break;
+ default:
+ goto error_unlock_exit;
+ }
+ if (ret < 0)
+ goto error_unlock_exit;
+ RTE_LOG(DEBUG, MEMBER, "Creating a setsummary table with mode %u\n",
+ setsum->type);
+
+ te->data = (void *)setsum;
+ TAILQ_INSERT_TAIL(member_list, te, next);
+ rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+ return setsum;
+
+error_unlock_exit:
+ rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+ rte_member_free(setsum);
+ return NULL;
+}
+
+
+int
+rte_member_add(const void *ss, const void *key, MEMBER_SET_TYPE set_id)
+{
+ const struct rte_member_setsum *setsum = ss;
+
+ if (setsum == NULL || key == NULL)
+ return -EINVAL;
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ return rte_member_add_ht(setsum, key, set_id);
+ case RTE_MEMBER_TYPE_VBF:
+ return rte_member_add_vbf(setsum, key, set_id);
+ default:
+ return -EINVAL;
+ }
+}
+
+
+int
+rte_member_lookup(const void *ss, const void *key,
+ MEMBER_SET_TYPE *set_id)
+{
+ const struct rte_member_setsum *setsum = ss;
+
+ if (setsum == NULL || key == NULL || set_id == NULL)
+ return -EINVAL;
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ return rte_member_lookup_ht(setsum, key, set_id);
+ case RTE_MEMBER_TYPE_VBF:
+ return rte_member_lookup_vbf(setsum, key, set_id);
+ default:
+ return -EINVAL;
+ }
+}
+
+
+int
+rte_member_lookup_bulk(const void *ss, const void **keys, uint32_t num_keys,
+ MEMBER_SET_TYPE *set_ids)
+{
+ const struct rte_member_setsum *setsum = ss;
+
+ if (setsum == NULL || keys == NULL || set_ids == NULL)
+ return -EINVAL;
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ return rte_member_lookup_bulk_ht(setsum, keys, num_keys,
+ set_ids);
+ case RTE_MEMBER_TYPE_VBF:
+ return rte_member_lookup_bulk_vbf(setsum, keys, num_keys,
+ set_ids);
+ default:
+ return -EINVAL;
+ }
+}
+
+int
+rte_member_lookup_multi(const void *ss, const void *key,
+ uint32_t match_per_key, MEMBER_SET_TYPE *set_id)
+{
+ const struct rte_member_setsum *setsum = ss;
+
+ if (setsum == NULL || key == NULL || set_id == NULL)
+ return -EINVAL;
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ return rte_member_lookup_multi_ht(setsum, key, match_per_key,
+ set_id);
+ case RTE_MEMBER_TYPE_VBF:
+ return rte_member_lookup_multi_vbf(setsum, key, match_per_key,
+ set_id);
+ default:
+ return -EINVAL;
+ }
+}
+
+
+int
+rte_member_lookup_multi_bulk(const void *ss, const void **keys,
+ uint32_t num_keys, uint32_t max_match_per_key,
+ uint32_t *match_count, MEMBER_SET_TYPE *set_ids)
+{
+ const struct rte_member_setsum *setsum = ss;
+
+ if (setsum == NULL || keys == NULL || set_ids == NULL ||
+ match_count == NULL)
+ return -EINVAL;
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ return rte_member_lookup_multi_bulk_ht(setsum, keys, num_keys,
+ max_match_per_key, match_count, set_ids);
+ case RTE_MEMBER_TYPE_VBF:
+ return rte_member_lookup_multi_bulk_vbf(setsum, keys, num_keys,
+ max_match_per_key, match_count, set_ids);
+ default:
+ return -EINVAL;
+ }
+}
+
+
+int
+rte_member_delete(void *ss, const void *key, MEMBER_SET_TYPE set_id)
+{
+ struct rte_member_setsum *setsum = ss;
+
+ if (setsum == NULL || key == NULL)
+ return -EINVAL;
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ return rte_member_delete_ht(setsum, key, set_id);
+ case RTE_MEMBER_TYPE_VBF:
+ default:
+ return -EINVAL;
+ }
+}
+
+
+void
+rte_member_reset(const void *ss)
+{
+ const struct rte_member_setsum *setsum = ss;
+
+ if (setsum == NULL)
+ return;
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ rte_member_reset_ht(setsum);
+ return;
+ case RTE_MEMBER_TYPE_VBF:
+ rte_member_reset_vbf(setsum);
+ return;
+ default:
+ return;
+ }
+}
diff --git a/lib/librte_member/rte_member.h b/lib/librte_member/rte_member.h
new file mode 100644
index 0000000..de44b1b
--- /dev/null
+++ b/lib/librte_member/rte_member.h
@@ -0,0 +1,518 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+
+
+/**
+ * @file
+ *
+ * RTE Membership Library
+ *
+ * The Membership Library is an extension and generalization of a traditional
+ * filter (for example Bloom Filter) structure that has multiple usages in a
+ * variety of workloads and applications. The library is used to test if a key
+ * belongs to certain sets. Two types of such "set-summary" structures are
+ * implemented: hash-table based (HT) and vector bloom filter (vBF). For HT
+ * setsummary, two subtype or modes are available, cache and non-cache modes.
+ * The table below summarize some properties of the different implementations.
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ */
+
+
+/**
+ * <!--
+ * +==========+=====================+================+=========================+
+ * | type | vbf | HT-cache | HT-non-cache |
+ * +==========+=====================+==========================================+
+ * |structure | bloom-filter array | hash-table like without storing key |
+ * +----------+---------------------+------------------------------------------+
+ * |set id | limited by bf count | [1, 0x7fff] |
+ * | | up to 32. | |
+ * +----------+---------------------+------------------------------------------+
+ * |usages & | small set range, | can delete, | cache most recent keys, |
+ * |properties| user-specified | big set range, | have both false-positive|
+ * | | false-positive rate,| small false | and false-negative |
+ * | | no deletion support.| positive depend| depend on table size, |
+ * | | | on table size, | automatic overwritten. |
+ * | | | new key does | |
+ * | | | not overwrite | |
+ * | | | existing key. | |
+ * +----------+---------------------+----------------+-------------------------+
+ * -->
+ */
+
+
+#ifndef _RTE_MEMBER_H_
+#define _RTE_MEMBER_H_
+
+
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+#include <stdio.h>
+#include <stdint.h>
+#include <rte_hash_crc.h>
+
+/** The set ID type that stored internally in hash table based set summary. */
+typedef uint16_t MEMBER_SET_TYPE;
+/** Invalid set ID used to mean no match found. */
+#define RTE_MEMBER_NO_MATCH 0
+/** Maximum size of hash table that can be created. */
+#define RTE_MEMBER_ENTRIES_MAX (1 << 30)
+/** Maximum number of keys that can be searched as a bulk */
+#define RTE_MEMBER_LOOKUP_BULK_MAX 64
+/** Entry count per bucket in hash table based mode. */
+#define RTE_MEMBER_BUCKET_ENTRIES 16
+/** Maximum number of characters in setsum name. */
+#define RTE_MEMBER_NAMESIZE 32
+
+/** @internal Primary hash function to calculate 1st independent hash. */
+#define MEMBER_PRIM_HASH(key, key_len, seed) \
+ (uint32_t)(rte_hash_crc(key, key_len, seed))
+/** @internal Secondary hash function to calculate 2nd independent hash. */
+#define MEMBER_SEC_HASH(key, key_len, seed) \
+ (uint32_t)(rte_hash_crc(key, key_len, seed))
+
+
+struct rte_member_setsum;
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Parameter struct used to create set summary
+ */
+struct rte_member_parameters;
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Define different set summary types
+ */
+enum rte_member_setsum_type {
+ RTE_MEMBER_TYPE_HT = 0, /**< Hash table based set summary. */
+ RTE_MEMBER_TYPE_VBF, /**< vector of bloom filters. */
+ RTE_MEMBER_NUM_TYPE
+};
+
+/** @internal compare function for different arch. */
+enum rte_member_sig_compare_function {
+ RTE_MEMBER_COMPARE_SCALAR = 0,
+ RTE_MEMBER_COMPARE_AVX2,
+ RTE_MEMBER_COMPARE_NUM
+};
+
+/** @internal setsummary structure. */
+struct rte_member_setsum {
+ enum rte_member_setsum_type type;
+ const char *name;
+ uint32_t key_len;
+ uint32_t socket_id; /* NUMA Socket ID for memory. */
+ uint32_t prim_hash_seed;
+ uint32_t sec_hash_seed;
+
+
+ /* hash table based */
+ uint32_t bucket_cnt;
+ uint32_t bucket_mask;
+ uint8_t cache;
+ enum rte_member_sig_compare_function sig_cmp_fn;
+
+ /* vector bloom filter*/
+ uint32_t num_set;
+ uint32_t bits;
+ uint32_t bit_mask;
+ uint32_t num_hashes;
+ void *table;
+};
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Parameters used when create the set summary table. Currently user can
+ * specify two types of setsummary: HT based and vBF. For HT based, user can
+ * specify cache or non-cache mode. Here is a table to describe some differences
+ *
+ */
+struct rte_member_parameters {
+ const char *name; /**< Name of the hash. */
+
+ /**
+ * User to specify the type of the setsummary from one of
+ * rte_member_setsum_type.
+ *
+ * HT based setsummary is implemented like a hash table. User should use
+ * this type when there are many sets.
+ *
+ * vBF setsummary is a vector of bloom filters. It is used when number
+ * of sets is not big (less than 32 for current implementation).
+ */
+ enum rte_member_setsum_type type;
+ /**
+ * If it is HT based setsummary, user to specify the subtype or mode
+ * of the setsummary. It could be cache, or non-cache mode.
+ * Set iscache to be 1 if to use as cache mode.
+ *
+ * For cache mode, keys can be evicted out of the HT setsummary. Keys
+ * with the same signature and map to the same bucket
+ * will overwrite each other in the setsummary table.
+ * This mode is useful for the case that the set-summary only
+ * needs to keep record of the recently inserted keys. Both
+ * false-negative and false-positive could happen.
+ *
+ * For non-cache mode, keys cannot be evicted out of the cache. So for
+ * this mode the setsummary will become full eventually. Keys with the
+ * same signature but map to the same bucket will still occupy multiple
+ * entries. This mode does not give false-negative result.
+ */
+ uint8_t iscache;
+
+ /**
+ * For HT setsummary, num_keys equals to the number of entries of the
+ * table. When the number of keys that inserted to the HT setsummary
+ * approaches this number, eviction could happen. For cache mode,
+ * keys could be evicted out of the table. For non-cache mode, keys will
+ * be evicted to other buckets like cuckoo hash. The table will also
+ * likely to become full before the number of inserted keys equal to the
+ * total number of entries.
+ *
+ * For vBF, num_keys equal to the expected number of keys that will
+ * be inserted into the vBF. The implementation assumes the keys are
+ * evenly distributed to each BF in vBF. This is used to calculate the
+ * number of bits we need for each BF. User does not specify the size of
+ * each BF directly because the optimal size depends on the num_keys
+ * and false positive rate.
+ */
+ uint32_t num_keys;
+
+
+ /**
+ * The length of key is used for hash calculation. Since key is not
+ * stored in set-summary, large key does not require more memory space.
+ */
+ uint32_t key_len;
+
+
+ /**
+ * num_set is only relevant to vBF based setsummary.
+ * num_set is equal to the number of BFs in vBF. For current
+ * implementation, it only supports 1,2,4,8,16,32 BFs in one vBF set
+ * summary. If other number of sets are needed, for example 5, the user
+ * should allocate the minimum available value that larger than 5,
+ * which is 8.
+ */
+ uint32_t num_set;
+
+ /**
+ * false_postive_rate is only relevant to vBF based setsummary.
+ * false_postivie_rate is the user-defined false positive rate
+ * given expected number of inserted keys (num_keys). It is used to
+ * calculate the total number of bits for each BF, and the number of
+ * hash values used during lookup and insertion. For details please
+ * refer to vBF implementation and membership library documentation.
+ *
+ * HT setsummary's false positive rate is in the order of:
+ * false_pos = (1/bucket_count)*(1/2^16), since we use 16-bit signature.
+ * This is because two keys needs to map to same bucket and same
+ * signature to have a collision (false positive). bucket_count is equal
+ * to number of entries (num_keys) divided by entry count per bucket
+ * (RTE_MEMBER_BUCKET_ENTRIES). Thus, the false_postivie_rate is not
+ * directly set by users.
+ */
+ float false_positive_rate;
+
+ /**
+ * We use two seeds to calculate two independent hashes for each key.
+ *
+ * For HT type, one hash is used as signature, and the other is used
+ * for bucket location.
+ * For vBF type, these two hashes and their combinations are used as
+ * hash locations to index the bit array.
+ */
+ uint32_t prim_hash_seed;
+
+ /**
+ * The secondary seed should be a different value from the primary seed.
+ */
+ uint32_t sec_hash_seed;
+
+ int socket_id; /**< NUMA Socket ID for memory. */
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Find an existing set-summary and return a pointer to it.
+ *
+ * @param name
+ * Name of the set-summary
+ * @return
+ * Pointer to the set-summary or NULL if object not found
+ * with rte_errno set appropriately. Possible rte_errno values include:
+ * - ENOENT - value not available for return
+ */
+void *
+rte_member_find_existing(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Create set-summary (SS).
+ *
+ * @param params
+ * parameters to initialize the setsummary
+ * @return
+ * return the pointer to the setsummary
+ * return value is NULL if the creation failed
+ */
+
+void *
+rte_member_create(const struct rte_member_parameters *params);
+
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Lookup key in set-summary (SS).
+ * Single key lookup and return as soon as the first match found
+ * @param setsum
+ * pointer of a setsummary
+ * @param key
+ * pointer of the key that needs to lookup
+ * @param set_id
+ * output the set id matches the key
+ * @return
+ * return 1 for found a match and 0 for not found a match
+ */
+
+int
+rte_member_lookup(const void *setsum,
+ const void *key, MEMBER_SET_TYPE *set_id);
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * lookup bulk of keys in set-summary (SS).
+ * Each key lookup returns as soon as the first match found
+ * @param setsum
+ * Pointer of a setsummary
+ * @param keys
+ * Pointer of bulk of keys that to be lookup
+ * @param num_keys
+ * Number of keys that will be lookup
+ * @param set_ids
+ * Output set ids for all the keys to this array.
+ * User should preallocate array that can contain all results, which size is
+ * the num_keys.
+ * @return
+ * The number of keys that found a match
+ */
+
+
+int
+rte_member_lookup_bulk(const void *setsum,
+ const void **keys, uint32_t num_keys,
+ MEMBER_SET_TYPE *set_ids);
+
+
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Lookup a key in set-summary (SS) for multiple matches.
+ * The key lookup will find all matched entries (multiple match).
+ * Note that for cache mode of HT, each key can have at most one match. This is
+ * because keys with same signature that maps to same bucket will overwrite
+ * each other. So multi-match lookup should be used for vBF and non-cache HT.
+ * @param setsum
+ * pointer of a set-summary
+ * @param key
+ * The key that to be lookup
+ * @param max_match_per_key
+ * User specified maximum number of matches for each key. The function returns
+ * as soon as this number of matches found for the key.
+ * @param set_id
+ * Output set ids for all the matches of the key. User needs to preallocate
+ * the array that can contain max_match_per_key number of results.
+ * @return
+ * The number of matches that found for the key.
+ * For cache mode HT set-summary, the number should be at most 1
+ */
+
+int
+rte_member_lookup_multi(const void *setsum,
+ const void *key, uint32_t max_match_per_key,
+ MEMBER_SET_TYPE *set_id);
+
+
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Lookup a bulk of keys in set-summary (SS) for multiple matches each key.
+ * Each key lookup will find all matched entries (multiple match).
+ * Note that for cache mode HT, each key can have at most one match. So
+ * multi-match function is mainly used for vBF and non-cache mode HT.
+ * @param setsum
+ * pointer of a setsummary
+ * @param keys
+ * The keys that to be lookup
+ * @param num_keys
+ * The number of keys that will be lookup
+ * @param max_match_per_key
+ * The possible maximum number of matches for each key
+ * @param match_count
+ * Output the number of matches for each key in an array
+ * @param set_ids
+ * Return set ids for all the matches of all keys. User pass in a preallocated
+ * 2D array with first dimension as key index and second dimension as match
+ * index. For example set_ids[bulk_size][max_match_per_key]
+ * @return
+ * The number of keys that found one or more matches in the set-summary
+ */
+
+int
+rte_member_lookup_multi_bulk(const void *setsum,
+ const void **keys, uint32_t num_keys,
+ uint32_t max_match_per_key,
+ uint32_t *match_count,
+ MEMBER_SET_TYPE *set_ids);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Insert key into set-summary (SS).
+ *
+ * @param setsum
+ * pointer of a set-summary
+ * @param key
+ * the key that needs to be added
+ * @param set_id
+ * The set id associated with the key that needs to be added. Different mode
+ * supports different set_id ranges. 0 cannot be used as set_id since
+ * RTE_MEMBER_NO_MATCH by default is set as 0.
+ * For HT mode, the set_id has range as [1, 0x7FFF], MSB is reserved.
+ * For vBF mode the set id is limited by the num_set parameter when create
+ * the set-summary.
+ * @return
+ * HT (cache mode) and vBF should never fail unless the set_id is not in the
+ * valid range. In such case -EINVAL is returned.
+ * For HT (non-cache mode) it could fail with -ENOSPC error code when table is
+ * full.
+ * For success it returns different values for different modes to provide
+ * extra information for users.
+ * Return 0 for HT (cache mode) if the add does not cause
+ * eviction, return 1 otherwise. Return 0 for HT mode if success, -ENOSPC for
+ * full, and 1 if cuckoo eviction happens. Always return 0 for vBF mode.
+ */
+
+int
+rte_member_add(const void *setsum, const void *key,
+ MEMBER_SET_TYPE set_id);
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * De-allocate memory used by set-summary.
+ * @param setsum
+ * Pointer to the set summary
+ */
+void
+rte_member_free(void *setsum);
+
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Reset the set-summary tables. E.g. reset bits to be 0 in BF,
+ * reset set_id in each entry to be RTE_MEMBER_NO_MATCH in HT based SS.
+ * @param setsum
+ * Pointer to the set-summary
+ */
+void
+rte_member_reset(const void *setsum);
+
+
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Delete items from the set-summary. Note that vBF does not support deletion
+ * in current implementation. For vBF, error code of -EINVAL will be returned.
+ * @param setsum
+ * Pointer to the set-summary
+ * @param key
+ * The key to be deleted
+ * @param set_id
+ * For HT mode, we need both key and its corresponding set_id to
+ * properly delete the key. Without set_id, we may delete other keys with the
+ * same signature
+ * @return
+ * If no entry found to delete, an error code of -ENOENT could be returned
+ */
+
+int
+rte_member_delete(void *setsum, const void *key,
+ MEMBER_SET_TYPE set_id);
+
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_MEMBER_H_ */
diff --git a/lib/librte_member/rte_member_version.map b/lib/librte_member/rte_member_version.map
new file mode 100644
index 0000000..a5877c9
--- /dev/null
+++ b/lib/librte_member/rte_member_version.map
@@ -0,0 +1,16 @@
+DPDK_17.11 {
+ global:
+
+ rte_member_create;
+ rte_member_find_existing;
+ rte_member_lookup;
+ rte_member_lookup_bulk;
+ rte_member_lookup_multi;
+ rte_member_lookup_multi_bulk;
+ rte_member_add;
+ rte_member_free;
+ rte_member_reset;
+ rte_member_delete;
+
+ local: *;
+};
--
2.7.4
Yipeng Wang
2017-09-02 01:24:36 UTC
Permalink
One of the set-summray structure is hash-table based
set-summary (HTSS). One example is cuckoo filter [1].

Comparing to a traditional hash table, HTSS has a much more
compact structure. For each element, only one signature and
its corresponding set ID is stored. No key comparison is required
during lookup. For the table structure, there are multiple entries
in each bucket, and the table is composed of many buckets.

Two modes are supported for HTSS, "cache" and "none-cache" modes.
The non-cache mode is more similar to traditional cuckoo filter.
When a bucket is full, one entry will be evicted to its
alternative bucket to make space for the new key. The table could
be full and then no more keys could be inserted. This mode has
false-positive rate but no false-negative. Multiple entries
with same signature could stay in the same bucket.

The "cache" mode does not evict key to its alternative bucket
when a bucket is full, an existing key will be evicted out of
the table like a cache. So the table will never reject keys when
it is full. Another property is in each bucket, there cannot be
multiple entries with same signature. The mode could have both
false-positive and false-negative probability.

This patch add the implementation of HTSS.

[1] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically
Better Than Bloom,” in Conference on emerging Networking
Experiments and Technologies, 2014.

Signed-off-by: Yipeng Wang <***@intel.com>
---
lib/librte_member/Makefile | 2 +-
lib/librte_member/rte_member_ht.c | 490 ++++++++++++++++++++++++++++++++++++++
lib/librte_member/rte_member_ht.h | 115 +++++++++
3 files changed, 606 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_member/rte_member_ht.c
create mode 100644 lib/librte_member/rte_member_ht.h

diff --git a/lib/librte_member/Makefile b/lib/librte_member/Makefile
index 1a79eaa..ad26548 100644
--- a/lib/librte_member/Makefile
+++ b/lib/librte_member/Makefile
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map
LIBABIVER := 1

# all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c
# install includes
SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h

diff --git a/lib/librte_member/rte_member_ht.c b/lib/librte_member/rte_member_ht.c
new file mode 100644
index 0000000..b2ae6d0
--- /dev/null
+++ b/lib/librte_member/rte_member_ht.c
@@ -0,0 +1,490 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <rte_errno.h>
+#include <rte_malloc.h>
+#include <rte_prefetch.h>
+#include <rte_random.h>
+#include <rte_log.h>
+
+#include "rte_member.h"
+#include "rte_member_ht.h"
+
+
+static inline int
+insert_overwrite_search(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ MEMBER_SET_TYPE set_id)
+{
+ int i;
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ if (buckets[bucket].sigs[i] == tmp_sig) {
+ buckets[bucket].sets[i] = set_id;
+ return 1;
+ }
+ }
+ return 0;
+}
+
+
+
+
+static inline int
+search_bucket_single(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ MEMBER_SET_TYPE *set_id)
+{
+ int iter;
+ for (iter = 0; iter < RTE_MEMBER_BUCKET_ENTRIES; iter++) {
+ if (tmp_sig == buckets[bucket].sigs[iter] &&
+ buckets[bucket].sets[iter] !=
+ RTE_MEMBER_NO_MATCH) {
+ *set_id = buckets[bucket].sets[iter];
+ return 1;
+ }
+ }
+ return 0;
+}
+
+
+static inline void
+search_bucket_multi(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ uint32_t *counter,
+ uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id)
+{
+ int iter;
+ for (iter = 0; iter < RTE_MEMBER_BUCKET_ENTRIES; iter++) {
+ if (tmp_sig == buckets[bucket].sigs[iter] &&
+ buckets[bucket].sets[iter] !=
+ RTE_MEMBER_NO_MATCH) {
+ set_id[*counter] = buckets[bucket].sets[iter];
+ (*counter)++;
+ if (*counter >= match_per_key)
+ return;
+ }
+ }
+}
+
+
+int
+rte_member_create_ht(struct rte_member_setsum *ss,
+ const struct rte_member_parameters *params)
+{
+ uint32_t i, j;
+ uint32_t size_bucket_t;
+ uint32_t num_entries = rte_align32pow2(params->num_keys);
+
+ if ((num_entries > RTE_MEMBER_ENTRIES_MAX) ||
+ !rte_is_power_of_2(RTE_MEMBER_BUCKET_ENTRIES) ||
+ num_entries < RTE_MEMBER_BUCKET_ENTRIES) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, MEMBER,
+ "Membership HT create with invalid parameters\n");
+ return -EINVAL;
+ }
+
+ uint32_t num_buckets = num_entries / RTE_MEMBER_BUCKET_ENTRIES;
+
+ size_bucket_t = sizeof(struct member_ht_bucket);
+
+ struct member_ht_bucket *buckets = rte_zmalloc_socket(NULL,
+ num_buckets * size_bucket_t,
+ RTE_CACHE_LINE_SIZE, ss->socket_id);
+
+ if (buckets == NULL)
+ return -ENOMEM;
+
+ ss->table = buckets;
+ ss->bucket_cnt = num_buckets;
+ ss->bucket_mask = num_buckets - 1;
+ ss->cache = params->iscache;
+
+ for (i = 0; i < num_buckets; i++) {
+ for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++)
+ buckets[i].sets[j] = RTE_MEMBER_NO_MATCH;
+ }
+
+
+ RTE_LOG(DEBUG, MEMBER, "Hash table based filter created, "
+ "the table has %u entries, %u buckets\n",
+ num_buckets,
+ num_buckets / RTE_MEMBER_BUCKET_ENTRIES);
+ return 0;
+}
+
+static inline
+void get_buckets_index(const struct rte_member_setsum *ss, const void *key,
+ uint32_t *prim_bkt, uint32_t *sec_bkt, SIG_TYPE *sig)
+{
+ uint32_t first_hash = MEMBER_PRIM_HASH(key, ss->key_len,
+ ss->prim_hash_seed);
+ uint32_t sec_hash = MEMBER_SEC_HASH(&first_hash, 4, ss->sec_hash_seed);
+ *sig = first_hash & SIG_BITMASK;
+ if (ss->cache) {
+ *prim_bkt = sec_hash & ss->bucket_mask;
+ *sec_bkt = (sec_hash >> 16) & ss->bucket_mask;
+ } else {
+ *prim_bkt = sec_hash & ss->bucket_mask;
+ *sec_bkt = (*prim_bkt ^ *sig) & ss->bucket_mask;
+ }
+}
+
+
+int
+rte_member_lookup_ht(const struct rte_member_setsum *ss,
+ const void *key, MEMBER_SET_TYPE *set_id)
+{
+ uint32_t prim_bucket, sec_bucket;
+ SIG_TYPE tmp_sig;
+ struct member_ht_bucket *buckets = ss->table;
+
+
+ *set_id = RTE_MEMBER_NO_MATCH;
+ get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
+
+ if (search_bucket_single(prim_bucket, tmp_sig, buckets,
+ set_id) ||
+ search_bucket_single(sec_bucket, tmp_sig,
+ buckets, set_id))
+ return 1;
+
+ return 0;
+}
+
+
+uint32_t
+rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss,
+ const void **keys, uint32_t num_keys, MEMBER_SET_TYPE *set_id)
+{
+ uint32_t i;
+ uint32_t ret = 0;
+ struct member_ht_bucket *buckets = ss->table;
+ SIG_TYPE tmp_sig[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+ uint32_t prim_buckets[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+ uint32_t sec_buckets[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+
+ for (i = 0; i < num_keys; i++) {
+ get_buckets_index(ss, keys[i], &prim_buckets[i],
+ &sec_buckets[i], &tmp_sig[i]);
+ rte_prefetch0(&buckets[prim_buckets[i]]);
+ rte_prefetch0(&buckets[sec_buckets[i]]);
+ }
+
+ for (i = 0; i < num_keys; i++) {
+ if (search_bucket_single(prim_buckets[i], tmp_sig[i],
+ buckets, &set_id[i]) ||
+ search_bucket_single(sec_buckets[i],
+ tmp_sig[i], buckets, &set_id[i]))
+ ret++;
+ else
+ set_id[i] = RTE_MEMBER_NO_MATCH;
+ }
+ return ret;
+}
+
+
+uint32_t
+rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,
+ const void *key, uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id)
+{
+ uint32_t ret = 0;
+ uint32_t prim_bucket, sec_bucket;
+ SIG_TYPE tmp_sig;
+ struct member_ht_bucket *buckets = ss->table;
+
+ get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
+
+ search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
+ match_per_key, set_id);
+ if (ret < match_per_key)
+ search_bucket_multi(sec_bucket, tmp_sig,
+ buckets, &ret, match_per_key, set_id);
+ return ret;
+}
+
+
+uint32_t
+rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,
+ const void **keys, uint32_t num_keys, uint32_t match_per_key,
+ uint32_t *match_count,
+ MEMBER_SET_TYPE *set_ids)
+{
+ uint32_t i;
+ uint32_t ret = 0;
+ struct member_ht_bucket *buckets = ss->table;
+ uint32_t match_cnt_t;
+ SIG_TYPE tmp_sig[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+ uint32_t prim_buckets[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+ uint32_t sec_buckets[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+
+ for (i = 0; i < num_keys; i++) {
+ get_buckets_index(ss, keys[i], &prim_buckets[i],
+ &sec_buckets[i], &tmp_sig[i]);
+ rte_prefetch0(&buckets[prim_buckets[i]]);
+ rte_prefetch0(&buckets[sec_buckets[i]]);
+ }
+ for (i = 0; i < num_keys; i++) {
+ match_cnt_t = 0;
+
+ search_bucket_multi(prim_buckets[i], tmp_sig[i],
+ buckets, &match_cnt_t, match_per_key,
+ &set_ids[i*match_per_key]);
+ if (match_cnt_t < match_per_key)
+ search_bucket_multi(sec_buckets[i], tmp_sig[i],
+ buckets, &match_cnt_t, match_per_key,
+ &set_ids[i*match_per_key]);
+ match_count[i] = match_cnt_t;
+ if (match_cnt_t != 0)
+ ret++;
+ }
+ return ret;
+}
+
+static inline int
+try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
+ SIG_TYPE sig, MEMBER_SET_TYPE set_id)
+{
+ int i;
+ /* If not full then insert into one slot*/
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ if (buckets[prim].sets[i] == RTE_MEMBER_NO_MATCH) {
+ buckets[prim].sigs[i] = sig;
+ buckets[prim].sets[i] = set_id;
+ return 0;
+ }
+ }
+ /* if prim failed, we need to access second cache line */
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ if (buckets[sec].sets[i] == RTE_MEMBER_NO_MATCH) {
+ buckets[sec].sigs[i] = sig;
+ buckets[sec].sets[i] = set_id;
+ return 0;
+ }
+ }
+ return -1;
+}
+
+
+static inline int
+try_overwrite(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
+ SIG_TYPE sig, MEMBER_SET_TYPE set_id)
+{
+ if (insert_overwrite_search(prim, sig, buckets, set_id) ||
+ insert_overwrite_search(sec, sig, buckets,
+ set_id))
+ return 0;
+ return -1;
+}
+
+static inline int
+evict_from_bucket(void)
+{
+ /* for now, we randomly pick one entry to evict */
+ return rte_rand() & (RTE_MEMBER_BUCKET_ENTRIES - 1);
+}
+
+
+
+/*
+ * This function is similar to the cuckoo hash make_space function in hash
+ * library
+ */
+static inline int
+make_space_bucket(const struct rte_member_setsum *ss, uint32_t bkt_num)
+{
+
+ static unsigned int nr_pushes;
+
+ unsigned int i, j;
+ int ret;
+ struct member_ht_bucket *buckets = ss->table;
+ uint32_t next_bucket_idx;
+ struct member_ht_bucket *next_bkt[RTE_MEMBER_BUCKET_ENTRIES];
+ struct member_ht_bucket *bkt = &buckets[bkt_num];
+ MEMBER_SET_TYPE flag_mask = 1U << (sizeof(MEMBER_SET_TYPE) * 8 - 1);
+ /*
+ * Push existing item (search for bucket with space in
+ * alternative locations) to its alternative location
+ */
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ /* Search for space in alternative locations */
+ next_bucket_idx = (bkt->sigs[i] ^ bkt_num) & ss->bucket_mask;
+ next_bkt[i] = &buckets[next_bucket_idx];
+ for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++) {
+ if (next_bkt[i]->sets[j] == RTE_MEMBER_NO_MATCH)
+ break;
+ }
+
+ if (j != RTE_MEMBER_BUCKET_ENTRIES)
+ break;
+ }
+
+ /* Alternative location has spare room (end of recursive function) */
+ if (i != RTE_MEMBER_BUCKET_ENTRIES) {
+ next_bkt[i]->sigs[j] = bkt->sigs[i];
+ next_bkt[i]->sets[j] = bkt->sets[i];
+ return i;
+ }
+
+ /* Pick entry that has not been pushed yet */
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++)
+ if ((bkt->sets[i] & flag_mask) == 0)
+ break;
+
+ /* All entries have been pushed, so entry cannot be added */
+ if (i == RTE_MEMBER_BUCKET_ENTRIES ||
+ nr_pushes > RTE_MEMBER_MAX_PUSHES)
+ return -ENOSPC;
+
+ next_bucket_idx = (bkt->sigs[i] ^ bkt_num) & ss->bucket_mask;
+ /* Set flag to indicate that this entry is going to be pushed */
+ bkt->sets[i] |= flag_mask;
+
+ nr_pushes++;
+ /* Need room in alternative bucket to insert the pushed entry */
+ ret = make_space_bucket(ss, next_bucket_idx);
+ /*
+ * After recursive function.
+ * Clear flags and insert the pushed entry
+ * in its alternative location if successful,
+ * or return error
+ */
+ bkt->sets[i] &= ~flag_mask;
+ nr_pushes = 0;
+ if (ret >= 0) {
+ next_bkt[i]->sigs[ret] = bkt->sigs[i];
+ next_bkt[i]->sets[ret] = bkt->sets[i];
+ return i;
+ } else
+ return ret;
+}
+
+
+int
+rte_member_add_ht(const struct rte_member_setsum *ss,
+ const void *key, MEMBER_SET_TYPE set_id)
+{
+ int ret;
+ uint32_t prim_bucket, sec_bucket;
+ SIG_TYPE tmp_sig;
+ struct member_ht_bucket *buckets = ss->table;
+ MEMBER_SET_TYPE flag_mask = 1U << (sizeof(MEMBER_SET_TYPE) * 8 - 1);
+
+ if (set_id == RTE_MEMBER_NO_MATCH || (set_id & flag_mask) != 0)
+ return -EINVAL;
+
+ get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
+
+ /* if it is cache based filter, we try overwriting existing entry */
+ if (ss->cache) {
+ ret = try_overwrite(buckets, prim_bucket, sec_bucket, tmp_sig,
+ set_id);
+ if (ret != -1)
+ return ret;
+ }
+ /* If not full then insert into one slot*/
+ ret = try_insert(buckets, prim_bucket, sec_bucket, tmp_sig, set_id);
+ if (ret != -1)
+ return ret;
+
+ /* random pick prim or sec for recursive displacement */
+
+ uint32_t select_bucket = (tmp_sig && 1U) ? prim_bucket : sec_bucket;
+ if (ss->cache) {
+ ret = evict_from_bucket();
+ buckets[select_bucket].sigs[ret] = tmp_sig;
+ buckets[select_bucket].sets[ret] = set_id;
+ return 1;
+ }
+
+ ret = make_space_bucket(ss, select_bucket);
+ if (ret >= 0) {
+ buckets[select_bucket].sigs[ret] = tmp_sig;
+ buckets[select_bucket].sets[ret] = set_id;
+ ret = 1;
+ }
+
+ return ret;
+}
+
+
+void
+rte_member_free_ht(struct rte_member_setsum *ss)
+{
+ rte_free(ss->table);
+}
+
+
+int
+rte_member_delete_ht(struct rte_member_setsum *ss, const void *key,
+ MEMBER_SET_TYPE set_id)
+{
+ int i;
+ uint32_t prim_bucket, sec_bucket;
+ SIG_TYPE tmp_sig;
+ struct member_ht_bucket *buckets = ss->table;
+
+ get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
+
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ if (tmp_sig == buckets[prim_bucket].sigs[i] &&
+ set_id == buckets[prim_bucket].sets[i]) {
+ buckets[prim_bucket].sets[i] = RTE_MEMBER_NO_MATCH;
+ return 0;
+ }
+ }
+
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ if (tmp_sig == buckets[sec_bucket].sigs[i] &&
+ set_id == buckets[sec_bucket].sets[i]) {
+ buckets[sec_bucket].sets[i] = RTE_MEMBER_NO_MATCH;
+ return 0;
+ }
+ }
+ return -ENOENT;
+}
+
+
+void
+rte_member_reset_ht(const struct rte_member_setsum *ss)
+{
+ uint32_t i, j;
+ struct member_ht_bucket *buckets = ss->table;
+ for (i = 0; i < ss->bucket_cnt; i++) {
+ for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++)
+ buckets[i].sets[j] = RTE_MEMBER_NO_MATCH;
+ }
+}
diff --git a/lib/librte_member/rte_member_ht.h b/lib/librte_member/rte_member_ht.h
new file mode 100644
index 0000000..aebf1db
--- /dev/null
+++ b/lib/librte_member/rte_member_ht.h
@@ -0,0 +1,115 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _RTE_MEMBER_HT_H_
+#define _RTE_MEMBER_HT_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* Maximum number of pushes for cuckoo path in HT mode. */
+#define RTE_MEMBER_MAX_PUSHES 100
+
+typedef uint16_t SIG_TYPE; /* signature size is 16 bit */
+#define SIG_BITMASK 0xFFFF /* signature mask, 16 bit */
+
+
+/* The bucket struct for ht setsum */
+struct member_ht_bucket {
+ SIG_TYPE sigs[RTE_MEMBER_BUCKET_ENTRIES]; /* 2-byte signature */
+ MEMBER_SET_TYPE sets[RTE_MEMBER_BUCKET_ENTRIES]; /* 2-byte set */
+} __rte_cache_aligned;
+
+
+
+int
+rte_member_create_ht(struct rte_member_setsum *ss,
+ const struct rte_member_parameters *params);
+
+int
+rte_member_lookup_ht(const struct rte_member_setsum *setsum,
+ const void *key, MEMBER_SET_TYPE *set_id);
+
+
+
+uint32_t
+rte_member_lookup_bulk_ht(const struct rte_member_setsum *setsum,
+ const void **keys, uint32_t num_keys,
+ MEMBER_SET_TYPE *set_ids);
+
+
+
+
+uint32_t
+rte_member_lookup_multi_ht(const struct rte_member_setsum *setsum,
+ const void *key, uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id);
+
+
+
+uint32_t
+rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *setsum,
+ const void **keys, uint32_t num_keys, uint32_t match_per_key,
+ uint32_t *match_count,
+ MEMBER_SET_TYPE *set_ids);
+
+
+
+
+int
+rte_member_add_ht(const struct rte_member_setsum *setsum,
+ const void *key, MEMBER_SET_TYPE set_id);
+
+void
+rte_member_free_ht(struct rte_member_setsum *setsum);
+
+
+
+
+int
+rte_member_delete_ht(struct rte_member_setsum *ss, const void *key,
+ MEMBER_SET_TYPE set_id);
+
+
+void
+rte_member_reset_ht(const struct rte_member_setsum *setsum);
+
+
+#ifdef __cplusplus
+}
+#endif
+
+
+
+#endif /* _RTE_MEMBER_HT_H_ */
--
2.7.4
Yipeng Wang
2017-09-02 01:24:37 UTC
Permalink
Bloom Filter (BF) [1] is a well-known space-efficient
probabilistic data structure that answers set membership queries.
Vector of Bloom Filters (vBF) is an extension to traditional BF
that supports multi-set membership testing. Traditional BF will
return found or not-found for each key. vBF will also return
which set the key belongs to if it is found.

Since each set requires a BF, vBF should be used when set count
is small. vBF's false positive rate could be set arbitrarily so
that its memory requirement and lookup speed is better in certain
cases comparing to HT based set-summary.

This patch adds the vBF implementation.

[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable
Errors,” Communications of the ACM, 1970.

Signed-off-by: Yipeng Wang <***@intel.com>
---
lib/librte_member/Makefile | 2 +-
lib/librte_member/rte_member_vbf.c | 350 +++++++++++++++++++++++++++++++++++++
lib/librte_member/rte_member_vbf.h | 85 +++++++++
3 files changed, 436 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_member/rte_member_vbf.c
create mode 100644 lib/librte_member/rte_member_vbf.h

diff --git a/lib/librte_member/Makefile b/lib/librte_member/Makefile
index ad26548..50275ed 100644
--- a/lib/librte_member/Makefile
+++ b/lib/librte_member/Makefile
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map
LIBABIVER := 1

# all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c rte_member_vbf.c
# install includes
SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h

diff --git a/lib/librte_member/rte_member_vbf.c b/lib/librte_member/rte_member_vbf.c
new file mode 100644
index 0000000..49ca69e
--- /dev/null
+++ b/lib/librte_member/rte_member_vbf.c
@@ -0,0 +1,350 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <math.h>
+#include <string.h>
+
+#include <rte_malloc.h>
+#include <rte_memory.h>
+#include <rte_errno.h>
+#include <rte_log.h>
+
+#include "rte_member.h"
+#include "rte_member_vbf.h"
+
+/*
+ * vBF currently implemented as a big array.
+ * The BFs have a vertical layout. Bits in same location of all bfs will stay
+ * in the same cache line.
+ * For example, if we have 32 bloom filters, we use a uint32_t array to
+ * represent all of them. array[0] represent the first location of all the
+ * bloom filters, array[1] represents the second location of all the
+ * bloom filters, etc. The advantage of this layout is to minimize the average
+ * number of memory accesses to test all bloom filters.
+ *
+ * Currently the implementation supports vBF containing 1,2,4,8,16,32 BFs.
+ */
+
+int
+rte_member_create_vbf(struct rte_member_setsum *ss,
+ const struct rte_member_parameters *params)
+{
+
+ if (params->num_set > 32 || !rte_is_power_of_2(params->num_set) ||
+ params->false_positive_rate == 0 ||
+ params->false_positive_rate > 1) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, MEMBER, "vBF create with invalid parameters\n");
+ return -EINVAL;
+ }
+
+ /* We assume expected keys evenly distribute to all BFs */
+ uint32_t num_keys_per_bf = (params->num_keys + ss->num_set - 1) /
+ ss->num_set;
+
+ if (num_keys_per_bf == 0) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, MEMBER, "BF expected number of key is zero\n");
+ return -EINVAL;
+ }
+ /*
+ * Note that the false positive rate is for all BFs in the vBF
+ * such that the single BF's false positive rate need to be
+ * calculated.
+ * Assume each BF's False positive rate is x. The total false positive
+ * rate is fp = 1-(1-x)^n.
+ * => x = 1 - (1-fp)^(1/n)
+ */
+
+ float x = 1 - pow((1 - params->false_positive_rate), 1.0 / ss->num_set);
+
+ if (x == 0) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, MEMBER, "BF false positive rate is zero\n");
+ return -EINVAL;
+ }
+
+ uint32_t bits = ceil((num_keys_per_bf *
+ log(x)) / log(1.0 / (pow(2.0, log(2.0)))));
+
+ /* We round to power of 2 for performance during lookup */
+ ss->bits = rte_align32pow2(bits);
+ if (ss->bits == 0) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, MEMBER, "BF size is zero\n");
+ return -EINVAL;
+ }
+
+ ss->num_hashes = (uint32_t)(log(2.0) * bits / num_keys_per_bf);
+ ss->bit_mask = ss->bits - 1;
+
+
+ /*
+ * Since we round the bits to power of 2, the final false positive
+ * rate will probably not be same as the user specified. We log the
+ * new value as debug message.
+ */
+ float new_fp = pow((1 - pow((1 - 1.0 / ss->bits), num_keys_per_bf *
+ ss->num_hashes)), ss->num_hashes);
+ new_fp = 1 - pow((1 - new_fp), ss->num_set);
+
+ /*
+ * reduce hash function count, until we approach the user specified
+ * false-positive rate. otherwise it is too conservative
+ */
+ int tmp_num_hash = ss->num_hashes;
+
+ while (tmp_num_hash > 1) {
+ float tmp_fp = new_fp;
+
+ tmp_num_hash--;
+ new_fp = pow((1 - pow((1 - 1.0 / ss->bits), num_keys_per_bf *
+ tmp_num_hash)), tmp_num_hash);
+ new_fp = 1 - pow((1 - new_fp), ss->num_set);
+
+ if (new_fp > params->false_positive_rate) {
+ new_fp = tmp_fp;
+ tmp_num_hash++;
+ break;
+ }
+ }
+
+ ss->num_hashes = tmp_num_hash;
+
+ RTE_LOG(DEBUG, MEMBER, "vector bloom filter created, "
+ "each bloom filter expects %u keys, needs %u bits, %u hashes, "
+ "with false positive rate set as %.5f, "
+ "The new calculated vBF false positive rate is %.5f\n",
+ num_keys_per_bf, ss->bits, ss->num_hashes, x, new_fp);
+
+ ss->table = rte_zmalloc_socket(NULL, sizeof(uint32_t) * ss->num_set *
+ (ss->bits >> 5), RTE_CACHE_LINE_SIZE, ss->socket_id);
+
+ if (ss->table == NULL)
+ return -ENOMEM;
+
+ return 0;
+}
+
+
+/*
+ * a is how many bits could be represented by one uint32_t variable.
+ * b is used for division shift
+ * shift is used for multiplication shift
+ */
+static inline uint32_t
+test_bit(uint32_t h1, uint32_t h2, uint32_t iter, uint32_t shift, uint32_t a,
+ uint32_t b, const struct rte_member_setsum *ss)
+{
+ uint32_t *vbf = ss->table;
+ uint32_t n = ss->num_set;
+ uint32_t bit_loc = (h1 + iter * h2) & ss->bit_mask;
+ /*
+ * x>>b is the divide, x & (a-1) is the mod, & (1<<n-1) to mask out bits
+ * we do not need
+ */
+ return (vbf[bit_loc>>b] >> ((bit_loc & (a - 1)) << shift)) &
+ ((1ULL << n) - 1);
+}
+
+
+static inline void
+set_bit(uint32_t h1, uint32_t h2, uint32_t iter, uint32_t shift, uint32_t a,
+ uint32_t b, const struct rte_member_setsum *ss, int32_t set)
+{
+ uint32_t *vbf = ss->table;
+ uint32_t bit_loc = (h1 + iter * h2) & ss->bit_mask;
+ vbf[bit_loc>>b] |= 1U << (((bit_loc & (a - 1)) << shift) + set - 1);
+}
+
+
+
+int
+rte_member_lookup_vbf(const struct rte_member_setsum *ss, const void *key,
+ MEMBER_SET_TYPE *set_id)
+{
+ uint32_t j;
+ uint32_t h1 = MEMBER_PRIM_HASH(key, ss->key_len, ss->prim_hash_seed);
+ uint32_t h2 = MEMBER_SEC_HASH(&h1, 4, ss->sec_hash_seed);
+ uint32_t mask = 0xFFFFFFFF;
+ uint32_t shift = __builtin_ctz(ss->num_set);
+ uint32_t a = 32 >> shift;
+ uint32_t b = __builtin_ctz(a);
+
+ for (j = 0; j < ss->num_hashes; j++)
+ mask &= test_bit(h1, h2, j, shift, a, b, ss);
+
+ if (mask) {
+ *set_id = __builtin_ctz(mask) + 1;
+ return 1;
+ }
+
+ *set_id = RTE_MEMBER_NO_MATCH;
+ return 0;
+}
+
+
+
+uint32_t
+rte_member_lookup_bulk_vbf(const struct rte_member_setsum *ss,
+ const void **keys, uint32_t num_keys, MEMBER_SET_TYPE *set_ids)
+{
+ uint32_t i, k, h1, h2;
+ uint32_t ret = 0;
+ uint32_t mask;
+ uint32_t shift = __builtin_ctz(ss->num_set);
+ uint32_t a = 32 >> shift;
+ uint32_t b = __builtin_ctz(a);
+
+ for (i = 0; i < num_keys; i++) {
+ mask = 0xFFFFFFFF;
+ h1 = MEMBER_PRIM_HASH(keys[i], ss->key_len,
+ ss->prim_hash_seed);
+ h2 = MEMBER_SEC_HASH(&h1, 4, ss->sec_hash_seed);
+ for (k = 0; k < ss->num_hashes; k++)
+ mask &= test_bit(h1, h2, k, shift, a, b, ss);
+
+ if (mask) {
+ set_ids[i] = __builtin_ctz(mask) + 1;
+ ret++;
+ } else
+ set_ids[i] = RTE_MEMBER_NO_MATCH;
+ }
+ return ret;
+}
+
+
+
+uint32_t
+rte_member_lookup_multi_vbf(const struct rte_member_setsum *ss,
+ const void *key, uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id)
+{
+ uint32_t ret = 0;
+ uint32_t j;
+ uint32_t h1 = MEMBER_PRIM_HASH(key, ss->key_len, ss->prim_hash_seed);
+ uint32_t h2 = MEMBER_SEC_HASH(&h1, 4, ss->sec_hash_seed);
+ uint32_t mask = 0xFFFFFFFF;
+ uint32_t shift = __builtin_ctz(ss->num_set);
+ uint32_t a = 32 >> shift;
+ uint32_t b = __builtin_ctz(a);
+
+ for (j = 0; j < ss->num_hashes; j++)
+ mask &= test_bit(h1, h2, j, shift, a, b, ss);
+
+ while (mask) {
+ uint32_t loc = __builtin_ctz(mask);
+ set_id[ret] = loc + 1;
+ ret++;
+ if (ret >= match_per_key)
+ return ret;
+ mask &= ~(1U << loc);
+ }
+ return ret;
+}
+
+
+
+uint32_t
+rte_member_lookup_multi_bulk_vbf(const struct rte_member_setsum *ss,
+ const void **keys, uint32_t num_keys, uint32_t match_per_key,
+ uint32_t *match_count,
+ MEMBER_SET_TYPE *set_ids)
+{
+ uint32_t i, k, h1, h2;
+ uint32_t ret = 0;
+ uint32_t match_cnt_t;
+ uint32_t mask;
+ uint32_t shift = __builtin_ctz(ss->num_set);
+ uint32_t a = 32 >> shift;
+ uint32_t b = __builtin_ctz(a);
+
+ for (i = 0; i < num_keys; i++) {
+ match_cnt_t = 0;
+ mask = 0xFFFFFFFF;
+ h1 = MEMBER_PRIM_HASH(keys[i], ss->key_len,
+ ss->prim_hash_seed);
+ h2 = MEMBER_SEC_HASH(&h1, 4, ss->sec_hash_seed);
+
+ for (k = 0; k < ss->num_hashes; k++)
+ mask &= test_bit(h1, h2, k, shift, a, b, ss);
+
+ while (mask) {
+ uint32_t loc = __builtin_ctz(mask);
+ if (match_cnt_t >= match_per_key)
+ break;
+ set_ids[i * match_per_key + match_cnt_t] = loc + 1;
+ match_cnt_t++;
+ mask &= ~(1U << loc);
+ }
+ match_count[i] = match_cnt_t;
+ if (match_cnt_t != 0)
+ ret++;
+ }
+ return ret;
+}
+
+
+int
+rte_member_add_vbf(const struct rte_member_setsum *ss,
+ const void *key, MEMBER_SET_TYPE set_id)
+{
+ uint32_t i, h1, h2;
+
+ if (set_id > ss->num_set || set_id == RTE_MEMBER_NO_MATCH)
+ return -EINVAL;
+
+ h1 = MEMBER_PRIM_HASH(key, ss->key_len, ss->prim_hash_seed);
+ h2 = MEMBER_SEC_HASH(&h1, 4, ss->sec_hash_seed);
+ uint32_t shift = __builtin_ctz(ss->num_set);
+ uint32_t a = 32 >> shift;
+ uint32_t b = __builtin_ctz(a);
+
+ for (i = 0; i < ss->num_hashes; i++)
+ set_bit(h1, h2, i, shift, a, b, ss, set_id);
+ return 0;
+}
+
+void
+rte_member_free_vbf(struct rte_member_setsum *ss)
+{
+ rte_free(ss->table);
+}
+
+
+void
+rte_member_reset_vbf(const struct rte_member_setsum *ss)
+{
+ uint32_t *vbf = ss->table;
+ memset(vbf, 0, (ss->num_set * ss->bits) >> 3);
+}
diff --git a/lib/librte_member/rte_member_vbf.h b/lib/librte_member/rte_member_vbf.h
new file mode 100644
index 0000000..3d3ee33
--- /dev/null
+++ b/lib/librte_member/rte_member_vbf.h
@@ -0,0 +1,85 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _RTE_MEMBER_VBF_H_
+#define _RTE_MEMBER_VBF_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+int
+rte_member_create_vbf(struct rte_member_setsum *ss,
+ const struct rte_member_parameters *params);
+
+
+int
+rte_member_lookup_vbf(const struct rte_member_setsum *setsum,
+ const void *key, MEMBER_SET_TYPE *set_id);
+
+
+uint32_t
+rte_member_lookup_bulk_vbf(const struct rte_member_setsum *setsum,
+ const void **keys, uint32_t num_keys,
+ MEMBER_SET_TYPE *set_ids);
+
+
+uint32_t
+rte_member_lookup_multi_vbf(const struct rte_member_setsum *setsum,
+ const void *key, uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id);
+
+
+uint32_t
+rte_member_lookup_multi_bulk_vbf(const struct rte_member_setsum *setsum,
+ const void **keys, uint32_t num_keys, uint32_t match_per_key,
+ uint32_t *match_count,
+ MEMBER_SET_TYPE *set_ids);
+
+int
+rte_member_add_vbf(const struct rte_member_setsum *setsum,
+ const void *key, MEMBER_SET_TYPE set_id);
+
+void
+rte_member_free_vbf(struct rte_member_setsum *ss);
+
+
+void
+rte_member_reset_vbf(const struct rte_member_setsum *setsum);
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_MEMBER_VBF_H_ */
--
2.7.4
Yipeng Wang
2017-09-02 01:24:38 UTC
Permalink
For key search, the signatures of all entries are compared against
the signature of the key that is being looked up. Since all
signatures are contguously put in a bucket, they can be compared
with vector instructions (AVX2), achieving higher lookup performance.

This patch adds AVX2 implementation in a separate header file.

Signed-off-by: Yipeng Wang <***@intel.com>
---
lib/librte_member/rte_member_ht.c | 143 ++++++++++++++++++++++++++++---------
lib/librte_member/rte_member_x86.h | 111 ++++++++++++++++++++++++++++
2 files changed, 222 insertions(+), 32 deletions(-)
create mode 100644 lib/librte_member/rte_member_x86.h

diff --git a/lib/librte_member/rte_member_ht.c b/lib/librte_member/rte_member_ht.c
index b2ae6d0..15e2534 100644
--- a/lib/librte_member/rte_member_ht.c
+++ b/lib/librte_member/rte_member_ht.c
@@ -40,6 +40,10 @@
#include "rte_member.h"
#include "rte_member_ht.h"

+#if defined(RTE_ARCH_X86)
+#include "rte_member_x86.h"
+#endif
+

static inline int
insert_overwrite_search(uint32_t bucket, SIG_TYPE tmp_sig,
@@ -135,6 +139,13 @@ rte_member_create_ht(struct rte_member_setsum *ss,
for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++)
buckets[i].sets[j] = RTE_MEMBER_NO_MATCH;
}
+#if defined(RTE_ARCH_X86)
+ if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2) &&
+ RTE_MEMBER_BUCKET_ENTRIES == 16)
+ ss->sig_cmp_fn = RTE_MEMBER_COMPARE_AVX2;
+ else
+#endif
+ ss->sig_cmp_fn = RTE_MEMBER_COMPARE_SCALAR;


RTE_LOG(DEBUG, MEMBER, "Hash table based filter created, "
@@ -174,11 +185,23 @@ rte_member_lookup_ht(const struct rte_member_setsum *ss,
*set_id = RTE_MEMBER_NO_MATCH;
get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);

- if (search_bucket_single(prim_bucket, tmp_sig, buckets,
- set_id) ||
- search_bucket_single(sec_bucket, tmp_sig,
- buckets, set_id))
- return 1;
+ switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+ case RTE_MEMBER_COMPARE_AVX2:
+ if (search_bucket_single_avx(prim_bucket, tmp_sig, buckets,
+ set_id) ||
+ search_bucket_single_avx(sec_bucket, tmp_sig,
+ buckets, set_id))
+ return 1;
+ break;
+#endif
+ default:
+ if (search_bucket_single(prim_bucket, tmp_sig, buckets,
+ set_id) ||
+ search_bucket_single(sec_bucket, tmp_sig,
+ buckets, set_id))
+ return 1;
+ }

return 0;
}
@@ -203,13 +226,27 @@ rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss,
}

for (i = 0; i < num_keys; i++) {
- if (search_bucket_single(prim_buckets[i], tmp_sig[i],
- buckets, &set_id[i]) ||
- search_bucket_single(sec_buckets[i],
- tmp_sig[i], buckets, &set_id[i]))
- ret++;
- else
- set_id[i] = RTE_MEMBER_NO_MATCH;
+ switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+ case RTE_MEMBER_COMPARE_AVX2:
+ if (search_bucket_single_avx(prim_buckets[i],
+ tmp_sig[i], buckets, &set_id[i]) ||
+ search_bucket_single_avx(sec_buckets[i],
+ tmp_sig[i], buckets, &set_id[i]))
+ ret++;
+ else
+ set_id[i] = RTE_MEMBER_NO_MATCH;
+ break;
+#endif
+ default:
+ if (search_bucket_single(prim_buckets[i], tmp_sig[i],
+ buckets, &set_id[i]) ||
+ search_bucket_single(sec_buckets[i],
+ tmp_sig[i], buckets, &set_id[i]))
+ ret++;
+ else
+ set_id[i] = RTE_MEMBER_NO_MATCH;
+ }
}
return ret;
}
@@ -227,12 +264,24 @@ rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,

get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);

- search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
- match_per_key, set_id);
- if (ret < match_per_key)
- search_bucket_multi(sec_bucket, tmp_sig,
- buckets, &ret, match_per_key, set_id);
- return ret;
+ switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+ case RTE_MEMBER_COMPARE_AVX2:
+ search_bucket_multi_avx(prim_bucket, tmp_sig, buckets,
+ &ret, match_per_key, set_id);
+ if (ret < match_per_key)
+ search_bucket_multi_avx(sec_bucket, tmp_sig,
+ buckets, &ret, match_per_key, set_id);
+ return ret;
+#endif
+ default:
+ search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
+ match_per_key, set_id);
+ if (ret < match_per_key)
+ search_bucket_multi(sec_bucket, tmp_sig,
+ buckets, &ret, match_per_key, set_id);
+ return ret;
+ }
}


@@ -259,16 +308,34 @@ rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,
for (i = 0; i < num_keys; i++) {
match_cnt_t = 0;

- search_bucket_multi(prim_buckets[i], tmp_sig[i],
- buckets, &match_cnt_t, match_per_key,
- &set_ids[i*match_per_key]);
- if (match_cnt_t < match_per_key)
- search_bucket_multi(sec_buckets[i], tmp_sig[i],
+ switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+ case RTE_MEMBER_COMPARE_AVX2:
+ search_bucket_multi_avx(prim_buckets[i], tmp_sig[i],
buckets, &match_cnt_t, match_per_key,
&set_ids[i*match_per_key]);
- match_count[i] = match_cnt_t;
- if (match_cnt_t != 0)
- ret++;
+ if (match_cnt_t < match_per_key)
+ search_bucket_multi_avx(sec_buckets[i],
+ tmp_sig[i], buckets, &match_cnt_t,
+ match_per_key,
+ &set_ids[i*match_per_key]);
+ match_count[i] = match_cnt_t;
+ if (match_cnt_t != 0)
+ ret++;
+ break;
+#endif
+ default:
+ search_bucket_multi(prim_buckets[i], tmp_sig[i],
+ buckets, &match_cnt_t, match_per_key,
+ &set_ids[i*match_per_key]);
+ if (match_cnt_t < match_per_key)
+ search_bucket_multi(sec_buckets[i], tmp_sig[i],
+ buckets, &match_cnt_t, match_per_key,
+ &set_ids[i*match_per_key]);
+ match_count[i] = match_cnt_t;
+ if (match_cnt_t != 0)
+ ret++;
+ }
}
return ret;
}
@@ -300,12 +367,24 @@ try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,

static inline int
try_overwrite(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
- SIG_TYPE sig, MEMBER_SET_TYPE set_id)
+ SIG_TYPE sig, MEMBER_SET_TYPE set_id,
+ enum rte_member_sig_compare_function cmp_fn)
{
- if (insert_overwrite_search(prim, sig, buckets, set_id) ||
- insert_overwrite_search(sec, sig, buckets,
- set_id))
- return 0;
+ switch (cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+ case RTE_MEMBER_COMPARE_AVX2:
+ if (insert_overwrite_search_avx(prim, sig, buckets, set_id) ||
+ insert_overwrite_search_avx(sec, sig, buckets,
+ set_id))
+ return 0;
+ break;
+#endif
+ default:
+ if (insert_overwrite_search(prim, sig, buckets, set_id) ||
+ insert_overwrite_search(sec, sig, buckets,
+ set_id))
+ return 0;
+ }
return -1;
}

@@ -411,7 +490,7 @@ rte_member_add_ht(const struct rte_member_setsum *ss,
/* if it is cache based filter, we try overwriting existing entry */
if (ss->cache) {
ret = try_overwrite(buckets, prim_bucket, sec_bucket, tmp_sig,
- set_id);
+ set_id, ss->sig_cmp_fn);
if (ret != -1)
return ret;
}
diff --git a/lib/librte_member/rte_member_x86.h b/lib/librte_member/rte_member_x86.h
new file mode 100644
index 0000000..c55f128
--- /dev/null
+++ b/lib/librte_member/rte_member_x86.h
@@ -0,0 +1,111 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _RTE_MEMBER_X86_H_
+#define _RTE_MEMBER_X86_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <x86intrin.h>
+
+
+#if defined(RTE_MACHINE_CPUFLAG_AVX2)
+
+
+static inline int
+insert_overwrite_search_avx(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ MEMBER_SET_TYPE set_id)
+{
+ uint32_t hitmask = _mm256_movemask_epi8((__m256i)_mm256_cmpeq_epi16(
+ _mm256_load_si256((__m256i const *)buckets[bucket].sigs),
+ _mm256_set1_epi16(tmp_sig)));
+ if (hitmask) {
+ uint32_t hit_idx = __builtin_ctzl(hitmask) / 2;
+ buckets[bucket].sets[hit_idx] = set_id;
+ return 1;
+ }
+ return 0;
+}
+
+
+static inline int
+search_bucket_single_avx(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ MEMBER_SET_TYPE *set_id)
+{
+ uint32_t hitmask = _mm256_movemask_epi8((__m256i)_mm256_cmpeq_epi16(
+ _mm256_load_si256((__m256i const *)buckets[bucket].sigs),
+ _mm256_set1_epi16(tmp_sig)));
+ while (hitmask) {
+ uint32_t hit_idx = __builtin_ctzl(hitmask) / 2;
+ if (buckets[bucket].sets[hit_idx] != RTE_MEMBER_NO_MATCH) {
+ *set_id = buckets[bucket].sets[hit_idx];
+ return 1;
+ }
+ hitmask &= ~(3U << (hit_idx) * 2);
+ }
+ return 0;
+}
+
+static inline void
+search_bucket_multi_avx(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ uint32_t *counter,
+ uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id)
+{
+ uint32_t hitmask = _mm256_movemask_epi8((__m256i)_mm256_cmpeq_epi16(
+ _mm256_load_si256((__m256i const *)buckets[bucket].sigs),
+ _mm256_set1_epi16(tmp_sig)));
+ while (hitmask) {
+ uint32_t hit_idx = __builtin_ctzl(hitmask) / 2;
+ if (buckets[bucket].sets[hit_idx] != RTE_MEMBER_NO_MATCH) {
+ set_id[*counter] = buckets[bucket].sets[hit_idx];
+ (*counter)++;
+ if (*counter >= match_per_key)
+ return;
+ }
+ hitmask &= ~(3U << (hit_idx) * 2);
+ }
+}
+#endif
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_MEMBER_X86_H_ */
--
2.7.4
Yipeng Wang
2017-09-02 01:24:39 UTC
Permalink
This patch enables the Membership library.

Signed-off-by: Yipeng Wang <***@intel.com>
---
MAINTAINERS | 7 +++++++
config/common_base | 5 +++++
lib/librte_member/Makefile | 2 ++
mk/rte.app.mk | 2 ++
4 files changed, 16 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index a0cd75e..e372edf 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -734,6 +734,13 @@ F: test/test/test_meter.c
F: examples/qos_meter/
F: doc/guides/sample_app_ug/qos_metering.rst

+Membership - EXPERIMENTAL
+M: Yipeng Wang <***@intel.com>
+M: Sameh Gobriel <***@intel.com>
+F: lib/librte_member/
+F: doc/guides/prog_guide/member_lib.rst
+F: test/test/test_member*
+

Other libraries
---------------
diff --git a/config/common_base b/config/common_base
index 5e97a08..5e31ced 100644
--- a/config/common_base
+++ b/config/common_base
@@ -595,6 +595,11 @@ CONFIG_RTE_LIBRTE_HASH_DEBUG=n
CONFIG_RTE_LIBRTE_EFD=y

#
+# Compile librte_member
+#
+CONFIG_RTE_LIBRTE_MEMBER=y
+
+#
# Compile librte_jobstats
#
CONFIG_RTE_LIBRTE_JOBSTATS=y
diff --git a/lib/librte_member/Makefile b/lib/librte_member/Makefile
index 50275ed..3bac1d0 100644
--- a/lib/librte_member/Makefile
+++ b/lib/librte_member/Makefile
@@ -37,6 +37,8 @@ LIB = librte_member.a
CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3

+LDLIBS += -lm
+
EXPORT_MAP := rte_member_version.map

LIBABIVER := 1
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index c25fdd9..c79acf0 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -86,6 +86,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CFGFILE) += -lrte_cfgfile
_LDLIBS-y += --whole-archive

_LDLIBS-$(CONFIG_RTE_LIBRTE_HASH) += -lrte_hash
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lrte_member
_LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lrte_vhost
_LDLIBS-$(CONFIG_RTE_LIBRTE_KVARGS) += -lrte_kvargs
_LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF) += -lrte_mbuf
@@ -196,6 +197,7 @@ endif
_LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lm
_LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrt
_LDLIBS-$(CONFIG_RTE_LIBRTE_METER) += -lm
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lm
ifeq ($(CONFIG_RTE_LIBRTE_VHOST_NUMA),y)
_LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lnuma
endif
--
2.7.4
Yipeng Wang
2017-09-02 01:24:40 UTC
Permalink
This patch adds functional and performance tests for membership
library.

Signed-off-by: Yipeng Wang <***@intel.com>
---
test/test/Makefile | 3 +
test/test/test_member.c | 682 +++++++++++++++++++++++++++++++++++++++++++
test/test/test_member_perf.c | 643 ++++++++++++++++++++++++++++++++++++++++
3 files changed, 1328 insertions(+)
create mode 100644 test/test/test_member.c
create mode 100644 test/test/test_member_perf.c

diff --git a/test/test/Makefile b/test/test/Makefile
index 42d9a49..b61dde3 100644
--- a/test/test/Makefile
+++ b/test/test/Makefile
@@ -123,6 +123,9 @@ SRCS-y += test_logs.c
SRCS-y += test_memcpy.c
SRCS-y += test_memcpy_perf.c

+
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member_perf.c
SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd.c
SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd_perf.c

diff --git a/test/test/test_member.c b/test/test/test_member.c
new file mode 100644
index 0000000..8965e81
--- /dev/null
+++ b/test/test/test_member.c
@@ -0,0 +1,682 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+/* This test is for membership library's simple feature test */
+
+#include <rte_memcpy.h>
+#include <rte_malloc.h>
+#include <rte_member.h>
+#include <rte_byteorder.h>
+#include <rte_random.h>
+#include <rte_debug.h>
+#include <rte_ip.h>
+
+#include "test.h"
+
+void *setsum_ht;
+void *setsum_cache;
+void *setsum_vbf;
+
+/* 5-tuple key type */
+struct flow_key {
+ uint32_t ip_src;
+ uint32_t ip_dst;
+ uint16_t port_src;
+ uint16_t port_dst;
+ uint8_t proto;
+} __attribute__((packed));
+
+
+/* Keys used by unit test functions */
+static struct flow_key keys[5] = {
+ {
+ .ip_src = IPv4(0x03, 0x02, 0x01, 0x00),
+ .ip_dst = IPv4(0x07, 0x06, 0x05, 0x04),
+ .port_src = 0x0908,
+ .port_dst = 0x0b0a,
+ .proto = 0x0c,
+ },
+ {
+ .ip_src = IPv4(0x13, 0x12, 0x11, 0x10),
+ .ip_dst = IPv4(0x17, 0x16, 0x15, 0x14),
+ .port_src = 0x1918,
+ .port_dst = 0x1b1a,
+ .proto = 0x1c,
+ },
+ {
+ .ip_src = IPv4(0x23, 0x22, 0x21, 0x20),
+ .ip_dst = IPv4(0x27, 0x26, 0x25, 0x24),
+ .port_src = 0x2928,
+ .port_dst = 0x2b2a,
+ .proto = 0x2c,
+ },
+ {
+ .ip_src = IPv4(0x33, 0x32, 0x31, 0x30),
+ .ip_dst = IPv4(0x37, 0x36, 0x35, 0x34),
+ .port_src = 0x3938,
+ .port_dst = 0x3b3a,
+ .proto = 0x3c,
+ },
+ {
+ .ip_src = IPv4(0x43, 0x42, 0x41, 0x40),
+ .ip_dst = IPv4(0x47, 0x46, 0x45, 0x44),
+ .port_src = 0x4948,
+ .port_dst = 0x4b4a,
+ .proto = 0x4c,
+ }
+};
+
+uint32_t test_set[5] = {1, 2, 3, 4, 5};
+
+#define ITERATIONS 3
+#define KEY_SIZE 4
+
+#define MAX_ENTRIES (1 << 16)
+uint8_t gened_keys[MAX_ENTRIES][KEY_SIZE];
+
+static struct rte_member_parameters params = {
+ .num_keys = MAX_ENTRIES, /* Total hash table entries. */
+ .key_len = KEY_SIZE, /* Length of hash key. */
+
+ /*num_set and false_positive_rate only relevant to vBF setsum*/
+ .num_set = 32,
+ .false_positive_rate = 0.03,
+ .prim_hash_seed = 1,
+ .sec_hash_seed = 11,
+ .socket_id = 0 /* NUMA Socket ID for memory. */
+};
+
+
+/*
+ * Sequence of operations for find existing setsummary
+ *
+ * - create setsum
+ * - find existing setsum: hit
+ * - find non-existing setsum: miss
+ *
+ */
+static int
+test_member_find_existing(void)
+{
+ void *tmp_setsum = NULL, *result = NULL;
+ struct rte_member_parameters tmp_params = {
+ .name = "member_find_existing",
+ .num_keys = MAX_ENTRIES, /* Total hash table entries. */
+ .key_len = KEY_SIZE, /* Length of hash key. */
+ .type = RTE_MEMBER_TYPE_HT,
+ .num_set = 32,
+ .false_positive_rate = 0.03,
+ .prim_hash_seed = 1,
+ .sec_hash_seed = 11,
+ .socket_id = 0 /* NUMA Socket ID for memory. */
+ };
+
+ /* Create */
+ tmp_setsum = rte_member_create(&tmp_params);
+ TEST_ASSERT(tmp_setsum != NULL, "setsum creation failed");
+
+ /* Try to find existing hash table */
+ result = rte_member_find_existing("member_find_existing");
+ TEST_ASSERT(result == tmp_setsum, "could not find existing setsum");
+
+ /* Try to find non-existing hash table */
+ result = rte_member_find_existing("member_find_non_existing");
+ TEST_ASSERT(result == NULL, "found setsum that shouldn't exist");
+
+ /* Cleanup. */
+ rte_member_free(tmp_setsum);
+
+ return 0;
+}
+
+
+/*
+ * Test for bad creating parameters
+ */
+static int
+test_member_create_bad_param(void)
+{
+ void *bad_setsum = NULL;
+ struct rte_member_parameters bad_params = {
+ .num_keys = MAX_ENTRIES, /* Total hash table entries. */
+ .key_len = KEY_SIZE, /* Length of hash key. */
+ .type = RTE_MEMBER_TYPE_HT,
+ .num_set = 32,
+ .false_positive_rate = 0.03,
+ .prim_hash_seed = 1,
+ .sec_hash_seed = 11,
+ .socket_id = 0 /* NUMA Socket ID for memory. */
+ };
+
+ bad_params.name = "bad_param1";
+ bad_params.num_set = 0;
+ bad_params.type = RTE_MEMBER_TYPE_VBF;
+ /* test with 0 set for vBF should fail */
+ bad_setsum = rte_member_create(&bad_params);
+ if (bad_setsum != NULL) {
+ rte_member_free(bad_setsum);
+ printf("Impossible creating setsum successfully with invalid "
+ "number of set for vBF\n");
+ return -1;
+ }
+
+ bad_params.name = "bad_param2";
+ bad_params.false_positive_rate = 0;
+ bad_params.num_set = 32;
+ /* test with 0 false positive for vBF should fail */
+ bad_setsum = rte_member_create(&bad_params);
+ if (bad_setsum != NULL) {
+ rte_member_free(bad_setsum);
+ printf("Impossible creating setsum successfully with invalid "
+ "false positive rate for vBF\n");
+ return -1;
+ }
+
+ bad_params.name = "bad_param3";
+ bad_params.false_positive_rate = 0.03;
+ bad_params.num_keys = 31;
+ /* test with less than 1 key per BF for vBF should fail */
+ bad_setsum = rte_member_create(&bad_params);
+ if (bad_setsum != NULL) {
+ rte_member_free(bad_setsum);
+ printf("Impossible creating setsum successfully with invalid "
+ "num_keys for vBF\n");
+ return -1;
+ }
+
+ bad_params.name = "bad_param4";
+ bad_params.type = RTE_MEMBER_TYPE_HT;
+ bad_params.num_keys = RTE_MEMBER_BUCKET_ENTRIES / 2;
+ /* test with less than 1 bucket for HTSS should fail */
+ bad_setsum = rte_member_create(&bad_params);
+ if (bad_setsum != NULL) {
+ rte_member_free(bad_setsum);
+ printf("Impossible creating setsum successfully with too few "
+ "number of keys(entries) for HT\n");
+ return -1;
+ }
+
+ bad_params.name = "bad_param5";
+ bad_params.num_keys = RTE_MEMBER_ENTRIES_MAX + 1;
+ /* test with more than maximum entries for HTSS should fail */
+ bad_setsum = rte_member_create(&bad_params);
+ if (bad_setsum != NULL) {
+ rte_member_free(bad_setsum);
+ printf("Impossible creating setsum successfully with to many "
+ "number of keys(entries) for HT\n");
+ return -1;
+ }
+
+ bad_params.name = "bad_param5";
+ /* test with same name should fail */
+ bad_setsum = rte_member_create(&bad_params);
+ if (bad_setsum != NULL) {
+ rte_member_free(bad_setsum);
+ printf("Impossible creating setsum successfully with existed "
+ "name\n");
+ return -1;
+ }
+
+ rte_member_free(bad_setsum);
+ return 0;
+}
+
+
+/* Create test setsummaries. */
+static int test_member_create(void)
+{
+ params.key_len = sizeof(struct flow_key);
+
+ params.name = "test_member_ht";
+ params.iscache = 0;
+ params.type = RTE_MEMBER_TYPE_HT;
+ setsum_ht = rte_member_create(&params);
+
+ params.name = "test_member_cache";
+ params.iscache = 1;
+ setsum_cache = rte_member_create(&params);
+
+ params.name = "test_member_vbf";
+ params.type = RTE_MEMBER_TYPE_VBF;
+ setsum_vbf = rte_member_create(&params);
+
+ if (setsum_ht == NULL || setsum_cache == NULL || setsum_vbf == NULL) {
+ printf("Creation of setsums fail\n");
+ return -1;
+ }
+ printf("Creation of setsums success\n");
+ return 0;
+}
+
+static int test_member_insert(void)
+{
+ int ret_ht, ret_cache, ret_vbf, i;
+
+ for (i = 0; i < 5; i++) {
+ ret_ht = rte_member_add(setsum_ht, &keys[i], test_set[i]);
+ ret_cache = rte_member_add(setsum_cache, &keys[i],
+ test_set[i]);
+ ret_vbf = rte_member_add(setsum_vbf, &keys[i], test_set[i]);
+ TEST_ASSERT(ret_ht >= 0 && ret_cache >= 0 && ret_vbf >= 0,
+ "insert error");
+ }
+ printf("insert key success\n");
+ return 0;
+}
+
+static int test_member_lookup(void)
+{
+ int ret_ht, ret_cache, ret_vbf, i;
+ uint16_t set_ht, set_cache, set_vbf;
+ MEMBER_SET_TYPE set_ids_ht[5] = {0};
+ MEMBER_SET_TYPE set_ids_cache[5] = {0};
+ MEMBER_SET_TYPE set_ids_vbf[5] = {0};
+
+ uint32_t num_key_ht = 5;
+ uint32_t num_key_cache = 5;
+ uint32_t num_key_vbf = 5;
+
+ const void *key_array[5];
+
+ /* single lookup test */
+ for (i = 0; i < 5; i++) {
+ ret_ht = rte_member_lookup(setsum_ht, &keys[i], &set_ht);
+ ret_cache = rte_member_lookup(setsum_cache, &keys[i],
+ &set_cache);
+ ret_vbf = rte_member_lookup(setsum_vbf, &keys[i], &set_vbf);
+ TEST_ASSERT(ret_ht >= 0 && ret_cache >= 0 && ret_vbf >= 0,
+ "single lookup function error");
+
+ TEST_ASSERT(set_ht == test_set[i] &&
+ set_cache == test_set[i] &&
+ set_vbf == test_set[i],
+ "single lookup set value error");
+ }
+ printf("lookup single key success\n");
+
+ /* bulk lookup test */
+ for (i = 0; i < 5; i++)
+ key_array[i] = &keys[i];
+
+ ret_ht = rte_member_lookup_bulk(setsum_ht, &key_array[0],
+ num_key_ht, set_ids_ht);
+
+ ret_cache = rte_member_lookup_bulk(setsum_cache, &key_array[0],
+ num_key_cache, set_ids_cache);
+
+ ret_vbf = rte_member_lookup_bulk(setsum_vbf, &key_array[0],
+ num_key_vbf, set_ids_vbf);
+
+ TEST_ASSERT(ret_ht >= 0 && ret_cache >= 0 && ret_vbf >= 0,
+ "bulk lookup function error");
+
+ for (i = 0; i < 5; i++) {
+ TEST_ASSERT((set_ids_ht[i] == test_set[i]) &&
+ (set_ids_cache[i] == test_set[i]) &&
+ (set_ids_vbf[i] == test_set[i]),
+ "bulk lookup result error");
+ }
+
+ return 0;
+}
+
+
+static int test_member_delete(void)
+{
+ int ret_ht, ret_cache, ret_vbf, i;
+ uint16_t set_ht, set_cache, set_vbf;
+ for (i = 0; i < 5; i++) {
+ ret_ht = rte_member_delete(setsum_ht, &keys[i], test_set[i]);
+ ret_cache = rte_member_delete(setsum_cache, &keys[i],
+ test_set[i]);
+ ret_vbf = rte_member_delete(setsum_vbf, &keys[i], test_set[i]);
+ /* VBF does not support delete yet, so return error code */
+ TEST_ASSERT(ret_ht >= 0 && ret_cache >= 0,
+ "key deletion function error");
+ TEST_ASSERT(ret_vbf < 0,
+ "vbf does not support deletion, error");
+ }
+
+ for (i = 0; i < 5; i++) {
+ ret_ht = rte_member_lookup(setsum_ht, &keys[i], &set_ht);
+ ret_cache = rte_member_lookup(setsum_cache, &keys[i],
+ &set_cache);
+ ret_vbf = rte_member_lookup(setsum_vbf, &keys[i], &set_vbf);
+ TEST_ASSERT(ret_ht >= 0 && ret_cache >= 0,
+ "key lookup function error");
+ TEST_ASSERT(set_ht == RTE_MEMBER_NO_MATCH &&
+ ret_cache == RTE_MEMBER_NO_MATCH,
+ "key deletion failed");
+ }
+ printf("delete success\n");
+ return 0;
+}
+
+
+static int test_member_multimatch(void)
+{
+ int ret_ht, ret_vbf, ret_cache;
+ MEMBER_SET_TYPE set_ids_ht[32] = {0};
+ MEMBER_SET_TYPE set_ids_vbf[32] = {0};
+ MEMBER_SET_TYPE set_ids_cache[32] = {0};
+
+ MEMBER_SET_TYPE set_ids_ht_m[5][32] = {{0} };
+ MEMBER_SET_TYPE set_ids_vbf_m[5][32] = {{0} };
+ MEMBER_SET_TYPE set_ids_cache_m[5][32] = {{0} };
+
+ uint32_t match_count_ht[5];
+ uint32_t match_count_vbf[5];
+ uint32_t match_count_cache[5];
+
+ uint32_t num_key_ht = 5;
+ uint32_t num_key_vbf = 5;
+ uint32_t num_key_cache = 5;
+
+ const void *key_array[5];
+
+ uint32_t i, j;
+ /* same key at most inserted 2*entry_per_bucket times for HT mode */
+ for (i = 1; i < 33; i++) {
+ for (j = 0; j < 5; j++) {
+ ret_ht = rte_member_add(setsum_ht, &keys[j], i);
+ ret_vbf = rte_member_add(setsum_vbf, &keys[j], i);
+ ret_cache = rte_member_add(setsum_cache, &keys[j], i);
+
+ TEST_ASSERT(ret_ht >= 0 && ret_vbf >= 0 &&
+ ret_cache >= 0,
+ "insert function error");
+ }
+ }
+
+ /* single multimatch test */
+ for (i = 0; i < 5; i++) {
+ ret_vbf = rte_member_lookup_multi(setsum_vbf, &keys[i], 32,
+ set_ids_vbf);
+ ret_ht = rte_member_lookup_multi(setsum_ht, &keys[i], 32,
+ set_ids_ht);
+ ret_cache = rte_member_lookup_multi(setsum_cache, &keys[i], 32,
+ set_ids_cache);
+ /*
+ * for cache mode, it does not support multimatch
+ * the mutimatch should work like single match
+ */
+ TEST_ASSERT(ret_ht == 32 && ret_vbf == 32 && ret_cache == 1,
+ "single lookup_multi error");
+ TEST_ASSERT(set_ids_cache[0] == 32,
+ "single lookup_multi cache error");
+
+ for (j = 1; j < 33; j++) {
+ TEST_ASSERT(set_ids_ht[j-1] == j &&
+ set_ids_vbf[j-1] == j,
+ "single multimatch lookup error");
+ }
+ }
+ printf("lookup single key for multimatch success\n");
+
+ /* bulk multimatch test */
+
+ for (i = 0; i < 5; i++)
+ key_array[i] = &keys[i];
+ ret_vbf = rte_member_lookup_multi_bulk(setsum_vbf,
+ &key_array[0], num_key_ht, 32, match_count_vbf,
+ (MEMBER_SET_TYPE *)set_ids_vbf_m);
+
+ ret_ht = rte_member_lookup_multi_bulk(setsum_ht,
+ &key_array[0], num_key_vbf, 32, match_count_ht,
+ (MEMBER_SET_TYPE *)set_ids_ht_m);
+
+ ret_cache = rte_member_lookup_multi_bulk(setsum_cache,
+ &key_array[0], num_key_cache, 32, match_count_cache,
+ (MEMBER_SET_TYPE *)set_ids_cache_m);
+
+
+ for (j = 0; j < 5; j++) {
+ TEST_ASSERT(match_count_ht[j] == 32,
+ "bulk multimatch lookup HT match count error");
+ TEST_ASSERT(match_count_vbf[j] == 32,
+ "bulk multimatch lookup vBF match count error");
+ TEST_ASSERT(match_count_cache[j] == 1,
+ "bulk multimatch lookup CACHE match count error");
+ TEST_ASSERT(set_ids_cache_m[j][0] == 32,
+ "bulk multimatch lookup CACHE set value error");
+
+ for (i = 1; i < 33; i++) {
+ TEST_ASSERT(set_ids_ht_m[j][i-1] == i,
+ "bulk multimatch lookup HT set value error");
+ TEST_ASSERT(set_ids_vbf_m[j][i-1] == i,
+ "bulk multimatch lookup vBF set value error");
+ }
+ }
+
+ printf("lookup for bulk multimatch success\n");
+
+ return 0;
+}
+
+
+static int key_compare(const void *key1, const void *key2)
+{
+ return memcmp(key1, key2, KEY_SIZE);
+}
+
+static void
+setup_keys_and_data(void)
+{
+ unsigned int i, j;
+ int num_duplicates;
+
+ /* Reset all arrays */
+ for (i = 0; i < KEY_SIZE; i++)
+ gened_keys[0][i] = 0;
+
+ /* Generate a list of keys, some of which may be duplicates */
+ for (i = 0; i < MAX_ENTRIES; i++) {
+ for (j = 0; j < KEY_SIZE; j++)
+ gened_keys[i][j] = rte_rand() & 0xFF;
+ }
+
+ /* Remove duplicates from the keys array */
+ do {
+ num_duplicates = 0;
+ /* Sort the list of keys to make it easier to find duplicates */
+ qsort(gened_keys, MAX_ENTRIES, KEY_SIZE, key_compare);
+
+ /* Sift through the list of keys and look for duplicates */
+ int num_duplicates = 0;
+ for (i = 0; i < MAX_ENTRIES - 1; i++) {
+ if (memcmp(gened_keys[i], gened_keys[i + 1],
+ KEY_SIZE) == 0) {
+ /* This key already exists, try again */
+ num_duplicates++;
+ for (j = 0; j < KEY_SIZE; j++)
+ gened_keys[i][j] = rte_rand() & 0xFF;
+ }
+ }
+ } while (num_duplicates != 0);
+}
+
+
+static inline int
+add_gened_keys(void *setsum, unsigned int *added_keys)
+{
+ int ret = 0;
+
+ for (*added_keys = 0; ret >= 0 && *added_keys < MAX_ENTRIES;
+ (*added_keys)++) {
+ uint16_t set = (rte_rand() & 0xf) + 1;
+ ret = rte_member_add(setsum, &gened_keys[*added_keys], set);
+ }
+ return ret;
+}
+
+
+static inline int
+add_gened_keys_cache(void *setsum, unsigned int *added_keys)
+{
+ int ret = 0;
+
+ for (*added_keys = 0; ret == 0 && *added_keys < MAX_ENTRIES;
+ (*added_keys)++) {
+ uint16_t set = (rte_rand() & 0xf) + 1;
+ ret = rte_member_add(setsum, &gened_keys[*added_keys], set);
+ }
+ return ret;
+}
+
+static int
+test_member_loadfactor(void)
+{
+ unsigned int j;
+ unsigned int added_keys, average_keys_added = 0;
+ int ret;
+
+ setup_keys_and_data();
+
+ rte_member_free(setsum_ht);
+ rte_member_free(setsum_cache);
+ rte_member_free(setsum_vbf);
+
+ params.key_len = KEY_SIZE;
+ params.name = "test_member_ht";
+ params.iscache = 0;
+ params.type = RTE_MEMBER_TYPE_HT;
+ setsum_ht = rte_member_create(&params);
+
+ params.name = "test_member_cache";
+ params.iscache = 1;
+ setsum_cache = rte_member_create(&params);
+
+
+ if (setsum_ht == NULL || setsum_cache == NULL) {
+ printf("Creation of setsums fail\n");
+ return -1;
+ }
+ /* test HT mode */
+ for (j = 0; j < ITERATIONS; j++) {
+ /* Add random entries until key cannot be added */
+ ret = add_gened_keys(setsum_ht, &added_keys);
+ if (ret != -ENOSPC) {
+ printf("Unexpected error when adding keys\n");
+ return -1;
+ }
+ average_keys_added += added_keys;
+
+ /* Reset the table */
+ rte_member_reset(setsum_ht);
+
+ /* Print a dot to show progress on operations */
+ printf(".");
+ fflush(stdout);
+ }
+
+ average_keys_added /= ITERATIONS;
+
+ printf("\nKeys inserted when no space(non-cache) = %.2f%% (%u/%u)\n",
+ ((double) average_keys_added / params.num_keys * 100),
+ average_keys_added, params.num_keys);
+
+ /* test cache mode */
+ added_keys = average_keys_added = 0;
+ for (j = 0; j < ITERATIONS; j++) {
+ /* Add random entries until key cannot be added */
+ ret = add_gened_keys_cache(setsum_cache, &added_keys);
+ if (ret != 1) {
+ printf("Unexpected error when adding keys\n");
+ return -1;
+ }
+ average_keys_added += added_keys;
+
+ /* Reset the table */
+ rte_member_reset(setsum_cache);
+
+ /* Print a dot to show progress on operations */
+ printf(".");
+ fflush(stdout);
+ }
+
+ average_keys_added /= ITERATIONS;
+
+ printf("\nKeys inserted when eviction happens(cache)= %.2f%% (%u/%u)\n",
+ ((double) average_keys_added / params.num_keys * 100),
+ average_keys_added, params.num_keys);
+ return 0;
+}
+
+static void
+perform_free(void)
+{
+ rte_member_free(setsum_ht);
+ rte_member_free(setsum_cache);
+ rte_member_free(setsum_vbf);
+}
+
+static int
+test_member(void)
+{
+ if (test_member_create_bad_param() < 0)
+ return -1;
+
+ if (test_member_find_existing() < 0)
+ return -1;
+
+ if (test_member_create() < 0) {
+ perform_free();
+ return -1;
+ }
+ if (test_member_insert() < 0) {
+ perform_free();
+ return -1;
+ }
+ if (test_member_lookup() < 0) {
+ perform_free();
+ return -1;
+ }
+ if (test_member_delete() < 0) {
+ perform_free();
+ return -1;
+ }
+ if (test_member_multimatch() < 0) {
+ perform_free();
+ return -1;
+ }
+ if (test_member_loadfactor() < 0) {
+ rte_member_free(setsum_ht);
+ rte_member_free(setsum_cache);
+ return -1;
+ }
+
+ perform_free();
+ return 0;
+}
+
+REGISTER_TEST_COMMAND(member_autotest, test_member);
diff --git a/test/test/test_member_perf.c b/test/test/test_member_perf.c
new file mode 100644
index 0000000..2bb3a31
--- /dev/null
+++ b/test/test/test_member_perf.c
@@ -0,0 +1,643 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <stdio.h>
+#include <inttypes.h>
+
+#include <rte_lcore.h>
+#include <rte_cycles.h>
+#include <rte_malloc.h>
+#include <rte_random.h>
+#include <rte_memcpy.h>
+#include <rte_thash.h>
+#include <rte_member.h>
+
+#include "test.h"
+
+#define NUM_KEYSIZES 10
+#define NUM_SHUFFLES 10
+#define MAX_KEYSIZE 64
+#define MAX_ENTRIES (1 << 19)
+#define KEYS_TO_ADD (MAX_ENTRIES * 75 / 100) /* 75% table utilization */
+#define NUM_LOOKUPS (KEYS_TO_ADD * 5) /* Loop among keys added, several times */
+#define VBF_SET_CNT 32
+#define BURST_SIZE 64
+#define VBF_FALSE_RATE 0.03
+
+
+static unsigned int test_socket_id;
+
+enum sstype {
+ HT = 0,
+ CACHE,
+ VBF,
+ NUM_TYPE
+};
+
+enum operations {
+ ADD = 0,
+ LOOKUP,
+ LOOKUP_BULK,
+ LOOKUP_MULTI,
+ LOOKUP_MULTI_BULK,
+ DELETE,
+ LOOKUP_MISS,
+ NUM_OPERATIONS
+};
+
+
+struct member_perf_params {
+ void *setsum[NUM_TYPE];
+ uint32_t key_size;
+ unsigned int cycle;
+};
+
+
+static uint32_t hashtest_key_lens[] = {
+ /* standard key sizes */
+ 4, 8, 16, 32, 48, 64,
+ /* IPv4 SRC + DST + protocol, unpadded */
+ 9,
+ /* IPv4 5-tuple, unpadded */
+ 13,
+ /* IPv6 5-tuple, unpadded */
+ 37,
+ /* IPv6 5-tuple, padded to 8-byte boundary */
+ 40
+};
+
+/* Array to store number of cycles per operation */
+uint64_t cycles[NUM_TYPE][NUM_KEYSIZES][NUM_OPERATIONS];
+uint64_t false_data[NUM_TYPE][NUM_KEYSIZES];
+uint64_t false_data_bulk[NUM_TYPE][NUM_KEYSIZES];
+uint64_t false_data_multi[NUM_TYPE][NUM_KEYSIZES];
+uint64_t false_data_multi_bulk[NUM_TYPE][NUM_KEYSIZES];
+
+uint64_t false_hit[NUM_TYPE][NUM_KEYSIZES];
+
+
+MEMBER_SET_TYPE data[NUM_TYPE][/* Array to store the data */KEYS_TO_ADD];
+
+/* Array to store all input keys */
+uint8_t keys[KEYS_TO_ADD][MAX_KEYSIZE];
+
+/* Shuffle the keys that have been added, so lookups will be totally random */
+static void
+shuffle_input_keys(struct member_perf_params *params)
+{
+ MEMBER_SET_TYPE temp_data;
+ unsigned int i, j;
+ uint32_t swap_idx;
+ uint8_t temp_key[MAX_KEYSIZE];
+
+ for (i = KEYS_TO_ADD - 1; i > 0; i--) {
+ swap_idx = rte_rand() % i;
+ memcpy(temp_key, keys[i], hashtest_key_lens[params->cycle]);
+ memcpy(keys[i], keys[swap_idx],
+ hashtest_key_lens[params->cycle]);
+ memcpy(keys[swap_idx], temp_key,
+ hashtest_key_lens[params->cycle]);
+ for (j = 0; j < NUM_TYPE; j++) {
+ temp_data = data[j][i];
+ data[j][i] = data[j][swap_idx];
+ data[j][swap_idx] = temp_data;
+ }
+ }
+}
+
+static int key_compare(const void *key1, const void *key2)
+{
+ return memcmp(key1, key2, MAX_KEYSIZE);
+}
+
+struct rte_member_parameters member_params = {
+ .num_keys = MAX_ENTRIES, /* Total hash table entries. */
+ .key_len = 4, /* Length of hash key. */
+
+ /* num_set and false_positive_rate only relevant to vBF setsum*/
+ .num_set = VBF_SET_CNT,
+ .false_positive_rate = 0.03,
+ .prim_hash_seed = 0,
+ .sec_hash_seed = 1,
+ .socket_id = 0, /* NUMA Socket ID for memory. */
+ };
+
+
+
+static int
+setup_keys_and_data(struct member_perf_params *params, unsigned int cycle,
+ int miss)
+{
+ unsigned int i, j;
+ int num_duplicates;
+
+ params->key_size = hashtest_key_lens[cycle];
+ params->cycle = cycle;
+
+ /* Reset all arrays */
+ for (i = 0; i < params->key_size; i++)
+ keys[0][i] = 0;
+
+ /* Generate a list of keys, some of which may be duplicates */
+ for (i = 0; i < KEYS_TO_ADD; i++) {
+ for (j = 0; j < params->key_size; j++)
+ keys[i][j] = rte_rand() & 0xFF;
+
+ data[HT][i] = data[CACHE][i] = (rte_rand() & 0x7FFE) + 1;
+ data[VBF][i] = rte_rand() % VBF_SET_CNT + 1;
+ }
+
+ /* Remove duplicates from the keys array */
+ do {
+ num_duplicates = 0;
+
+ /* Sort the list of keys to make it easier to find duplicates */
+ qsort(keys, KEYS_TO_ADD, MAX_KEYSIZE, key_compare);
+
+ /* Sift through the list of keys and look for duplicates */
+ int num_duplicates = 0;
+ for (i = 0; i < KEYS_TO_ADD - 1; i++) {
+ if (memcmp(keys[i], keys[i + 1],
+ params->key_size) == 0) {
+ /* This key already exists, try again */
+ num_duplicates++;
+ for (j = 0; j < params->key_size; j++)
+ keys[i][j] = rte_rand() & 0xFF;
+ }
+ }
+ } while (num_duplicates != 0);
+
+ /* Shuffle the random values again */
+ shuffle_input_keys(params);
+
+ /* For testing miss lookup, we insert half and lookup the other half */
+ unsigned int entry_cnt, bf_key_cnt;
+ if (!miss) {
+ entry_cnt = MAX_ENTRIES;
+ bf_key_cnt = KEYS_TO_ADD;
+ } else {
+ entry_cnt = MAX_ENTRIES / 2;
+ bf_key_cnt = KEYS_TO_ADD / 2;
+ }
+ member_params.false_positive_rate = VBF_FALSE_RATE;
+ member_params.key_len = params->key_size;
+ member_params.socket_id = test_socket_id;
+ member_params.num_keys = entry_cnt;
+ member_params.name = "test_member_ht";
+ member_params.iscache = 0;
+ member_params.type = RTE_MEMBER_TYPE_HT;
+ params->setsum[HT] = rte_member_create(&member_params);
+ if (params->setsum[HT] == NULL)
+ fprintf(stderr, "ht create fail\n");
+
+ member_params.name = "test_member_cache";
+ member_params.iscache = 1;
+ params->setsum[CACHE] = rte_member_create(&member_params);
+ if (params->setsum[CACHE] == NULL)
+ fprintf(stderr, "CACHE create fail\n");
+
+ member_params.name = "test_member_vbf";
+ member_params.type = RTE_MEMBER_TYPE_VBF;
+ member_params.num_keys = bf_key_cnt;
+ params->setsum[VBF] = rte_member_create(&member_params);
+ if (params->setsum[VBF] == NULL)
+ fprintf(stderr, "VBF create fail\n");
+ for (i = 0; i < NUM_TYPE; i++) {
+ if (params->setsum[i] == NULL)
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+timed_adds(struct member_perf_params *params, int type)
+{
+ const uint64_t start_tsc = rte_rdtsc();
+ unsigned int i, a;
+ int32_t ret;
+
+ for (i = 0; i < KEYS_TO_ADD; i++) {
+ ret = rte_member_add(params->setsum[type], &keys[i],
+ data[type][i]);
+ if (ret < 0) {
+ printf("Error %d in rte_member_add - key=0x", ret);
+ for (a = 0; a < params->key_size; a++)
+ printf("%02x", keys[i][a]);
+ printf(" value=%d, type: %d\n", data[type][i], type);
+
+ return -1;
+ }
+ }
+
+ const uint64_t end_tsc = rte_rdtsc();
+ const uint64_t time_taken = end_tsc - start_tsc;
+
+ cycles[type][params->cycle][ADD] = time_taken / KEYS_TO_ADD;
+ return 0;
+}
+
+static int
+timed_lookups(struct member_perf_params *params, int type)
+{
+ unsigned int i, j;
+
+ false_data[type][params->cycle] = 0;
+
+ const uint64_t start_tsc = rte_rdtsc();
+ MEMBER_SET_TYPE result;
+ int ret;
+
+ for (i = 0; i < NUM_LOOKUPS / KEYS_TO_ADD; i++) {
+ for (j = 0; j < KEYS_TO_ADD; j++) {
+ ret = rte_member_lookup(params->setsum[type], &keys[j],
+ &result);
+ if (ret < 0) {
+ printf("lookup wrong internally");
+ return -1;
+ }
+ if (result != data[type][j])
+ false_data[type][params->cycle]++;
+ }
+ }
+
+ const uint64_t end_tsc = rte_rdtsc();
+ const uint64_t time_taken = end_tsc - start_tsc;
+
+ cycles[type][params->cycle][LOOKUP] = time_taken / NUM_LOOKUPS;
+
+ return 0;
+}
+
+static int
+timed_lookups_bulk(struct member_perf_params *params, int type)
+{
+ unsigned int i, j, k;
+ MEMBER_SET_TYPE result[BURST_SIZE] = {0};
+ const void *keys_burst[BURST_SIZE];
+ int ret;
+
+ false_data_bulk[type][params->cycle] = 0;
+
+ const uint64_t start_tsc = rte_rdtsc();
+
+ for (i = 0; i < NUM_LOOKUPS / KEYS_TO_ADD; i++) {
+ for (j = 0; j < KEYS_TO_ADD / BURST_SIZE; j++) {
+ for (k = 0; k < BURST_SIZE; k++)
+ keys_burst[k] = keys[j * BURST_SIZE + k];
+
+ ret = rte_member_lookup_bulk(params->setsum[type],
+ &keys_burst[0],
+ BURST_SIZE,
+ result);
+ if (ret <= 0) {
+ printf("lookup bulk has wrong return value\n");
+ return -1;
+ }
+ for (k = 0; k < BURST_SIZE; k++) {
+ uint32_t data_idx = j * BURST_SIZE + k;
+ if (result[k] != data[type][data_idx])
+ false_data_bulk[type][params->cycle]++;
+ }
+ }
+ }
+
+ const uint64_t end_tsc = rte_rdtsc();
+ const uint64_t time_taken = end_tsc - start_tsc;
+
+ cycles[type][params->cycle][LOOKUP_BULK] = time_taken / NUM_LOOKUPS;
+
+ return 0;
+}
+
+
+static int
+timed_lookups_multimatch(struct member_perf_params *params, int type)
+{
+ unsigned int i, j;
+ MEMBER_SET_TYPE result[RTE_MEMBER_BUCKET_ENTRIES] = {0};
+ int ret;
+ false_data_multi[type][params->cycle] = 0;
+
+ const uint64_t start_tsc = rte_rdtsc();
+
+ for (i = 0; i < NUM_LOOKUPS / KEYS_TO_ADD; i++) {
+ for (j = 0; j < KEYS_TO_ADD; j++) {
+ ret = rte_member_lookup_multi(params->setsum[type],
+ &keys[j], RTE_MEMBER_BUCKET_ENTRIES, result);
+ if (type != CACHE && ret <= 0) {
+ printf("lookup multi has wrong return value %d,"
+ "type %d\n", ret, type);
+ }
+ if (result[0] != data[type][j])
+ false_data_multi[type][params->cycle]++;
+ }
+ }
+
+ const uint64_t end_tsc = rte_rdtsc();
+ const uint64_t time_taken = end_tsc - start_tsc;
+
+ cycles[type][params->cycle][LOOKUP_MULTI] = time_taken / NUM_LOOKUPS;
+
+ return 0;
+}
+
+
+static int
+timed_lookups_multimatch_bulk(struct member_perf_params *params, int type)
+{
+ unsigned int i, j, k;
+ MEMBER_SET_TYPE result[BURST_SIZE][RTE_MEMBER_BUCKET_ENTRIES] = {{0} };
+ const void *keys_burst[BURST_SIZE];
+ uint32_t match_count[BURST_SIZE];
+ int ret;
+
+ false_data_multi_bulk[type][params->cycle] = 0;
+
+ const uint64_t start_tsc = rte_rdtsc();
+
+ for (i = 0; i < NUM_LOOKUPS / KEYS_TO_ADD; i++) {
+ for (j = 0; j < KEYS_TO_ADD / BURST_SIZE; j++) {
+ for (k = 0; k < BURST_SIZE; k++)
+ keys_burst[k] = keys[j * BURST_SIZE + k];
+
+ ret = rte_member_lookup_multi_bulk(
+ params->setsum[type],
+ &keys_burst[0], BURST_SIZE,
+ RTE_MEMBER_BUCKET_ENTRIES, match_count,
+ (MEMBER_SET_TYPE *)result);
+ if (ret < 0) {
+ printf("lookup multimatch bulk has wrong return"
+ " value\n");
+ return -1;
+ }
+ for (k = 0; k < BURST_SIZE; k++) {
+ if (type != CACHE && match_count[k] == 0) {
+ printf("lookup multimatch bulk get "
+ "wrong match count\n");
+ return -1;
+ }
+ uint32_t data_idx = j * BURST_SIZE + k;
+ if (result[k][0] != data[type][data_idx])
+ false_data_multi_bulk[type][params->cycle]++;
+ }
+ }
+ }
+
+ const uint64_t end_tsc = rte_rdtsc();
+ const uint64_t time_taken = end_tsc - start_tsc;
+
+ cycles[type][params->cycle][LOOKUP_MULTI_BULK] = time_taken /
+ NUM_LOOKUPS;
+
+ return 0;
+}
+
+static int
+timed_deletes(struct member_perf_params *params, int type)
+{
+ unsigned int i;
+ int32_t ret;
+
+ if (type == VBF)
+ return 0;
+ const uint64_t start_tsc = rte_rdtsc();
+ for (i = 0; i < KEYS_TO_ADD; i++) {
+ ret = rte_member_delete(params->setsum[type], &keys[i],
+ data[type][i]);
+ if (type != CACHE && ret < 0) {
+ printf("delete error\n");
+ return -1;
+ }
+ }
+
+ const uint64_t end_tsc = rte_rdtsc();
+ const uint64_t time_taken = end_tsc - start_tsc;
+
+ cycles[type][params->cycle][DELETE] = time_taken / KEYS_TO_ADD;
+
+ return 0;
+}
+
+
+static int
+timed_miss_lookup(struct member_perf_params *params, int type)
+{
+ unsigned int i, j;
+ int ret;
+
+ false_hit[type][params->cycle] = 0;
+
+ for (i = 0; i < KEYS_TO_ADD / 2; i++) {
+ ret = rte_member_add(params->setsum[type], &keys[i],
+ data[type][i]);
+ if (ret < 0) {
+ unsigned int a;
+ printf("Error %d in rte_member_add - key=0x", ret);
+ for (a = 0; a < params->key_size; a++)
+ printf("%02x", keys[i][a]);
+ printf(" value=%d, type: %d\n", data[type][i], type);
+
+ return -1;
+ }
+ }
+
+ const uint64_t start_tsc = rte_rdtsc();
+ MEMBER_SET_TYPE result;
+
+ for (i = 0; i < 2 * NUM_LOOKUPS / KEYS_TO_ADD; i++) {
+ for (j = KEYS_TO_ADD / 2; j < KEYS_TO_ADD; j++) {
+ ret = rte_member_lookup(params->setsum[type], &keys[j],
+ &result);
+ if (ret < 0) {
+ printf("lookup wrong internally");
+ return -1;
+ }
+ if (result != RTE_MEMBER_NO_MATCH)
+ false_hit[type][params->cycle]++;
+ }
+ }
+
+ const uint64_t end_tsc = rte_rdtsc();
+ const uint64_t time_taken = end_tsc - start_tsc;
+
+ cycles[type][params->cycle][LOOKUP_MISS] = time_taken / NUM_LOOKUPS;
+
+ return 0;
+}
+
+
+
+static void
+perform_frees(struct member_perf_params *params)
+{
+ int i;
+ for (i = 0; i < NUM_TYPE; i++) {
+ if (params->setsum[i] != NULL) {
+ rte_member_free(params->setsum[i]);
+ params->setsum[i] = NULL;
+ }
+ }
+}
+
+static int
+exit_with_fail(const char *testname, struct member_perf_params *params,
+ unsigned int i, unsigned int j)
+{
+ printf("<<<<<Test %s failed at keysize %d iteration %d type %d>>>>>\n",
+ testname, hashtest_key_lens[params->cycle], i, j);
+ perform_frees(params);
+ return -1;
+}
+
+static int
+run_all_tbl_perf_tests(void)
+{
+ unsigned int i, j, k;
+ struct member_perf_params params;
+
+ printf("Measuring performance, please wait\n");
+ fflush(stdout);
+
+ test_socket_id = rte_socket_id();
+
+
+ for (i = 0; i < NUM_KEYSIZES; i++) {
+ if (setup_keys_and_data(&params, i, 0) < 0) {
+ printf("Could not create keys/data/table\n");
+ return -1;
+ }
+ for (j = 0; j < NUM_TYPE; j++) {
+
+ if (timed_adds(&params, j) < 0)
+ return exit_with_fail("timed_adds", &params,
+ i, j);
+
+ for (k = 0; k < NUM_SHUFFLES; k++)
+ shuffle_input_keys(&params);
+
+ if (timed_lookups(&params, j) < 0)
+ return exit_with_fail("timed_lookups", &params,
+ i, j);
+
+ if (timed_lookups_bulk(&params, j) < 0)
+ return exit_with_fail("timed_lookups_bulk",
+ &params, i, j);
+
+ if (timed_lookups_multimatch(&params, j) < 0)
+ return exit_with_fail("timed_lookups_multi",
+ &params, i, j);
+
+ if (timed_lookups_multimatch_bulk(&params, j) < 0)
+ return exit_with_fail("timed_lookups_multi_bulk",
+ &params, i, j);
+
+ if (timed_deletes(&params, j) < 0)
+ return exit_with_fail("timed_deletes", &params,
+ i, j);
+
+ /* Print a dot to show progress on operations */
+ }
+ printf(".");
+ fflush(stdout);
+
+ perform_frees(&params);
+ }
+
+ /* test false postivie rate using un-inserted keys */
+ for (i = 0; i < NUM_KEYSIZES; i++) {
+ if (setup_keys_and_data(&params, i, 1) < 0) {
+ printf("Could not create keys/data/table\n");
+ return -1;
+ }
+ for (j = 0; j < NUM_TYPE; j++) {
+ if (timed_miss_lookup(&params, j) < 0)
+ return exit_with_fail("timed_miss_lookup",
+ &params, i, j);
+ }
+ perform_frees(&params);
+ }
+
+ printf("\nResults (in CPU cycles/operation)\n");
+ printf("-----------------------------------\n");
+ printf("\n%-18s%-18s%-18s%-18s%-18s%-18s%-18s%-18s%-18s\n",
+ "Keysize", "type", "Add", "Lookup", "Lookup_bulk",
+ "lookup_multi", "lookup_multi_bulk", "Delete",
+ "miss_lookup");
+ for (i = 0; i < NUM_KEYSIZES; i++) {
+ for (j = 0; j < NUM_TYPE; j++) {
+ printf("%-18d", hashtest_key_lens[i]);
+ printf("%-18d", j);
+ for (k = 0; k < NUM_OPERATIONS; k++)
+ printf("%-18"PRIu64, cycles[j][i][k]);
+ printf("\n");
+ }
+ }
+
+ printf("\nFalse results rate (and false positive rate)\n");
+ printf("-----------------------------------\n");
+ printf("\n%-18s%-18s%-18s%-18s%-18s%-18s%-18s\n",
+ "Keysize", "type", "fr_single", "fr_bulk", "fr_multi",
+ "fr_multi_bulk", "false_positive_rate");
+ /* key size not influence False rate so just print out one key size */
+ for (i = 0; i < 1; i++) {
+ for (j = 0; j < NUM_TYPE; j++) {
+ printf("%-18d", hashtest_key_lens[i]);
+ printf("%-18d", j);
+ printf("%-18f", (float)false_data[j][i] / NUM_LOOKUPS);
+ printf("%-18f", (float)false_data_bulk[j][i] /
+ NUM_LOOKUPS);
+ printf("%-18f", (float)false_data_multi[j][i] /
+ NUM_LOOKUPS);
+ printf("%-18f", (float)false_data_multi_bulk[j][i] /
+ NUM_LOOKUPS);
+ printf("%-18f", (float)false_hit[j][i] /
+ NUM_LOOKUPS);
+ printf("\n");
+ }
+ }
+
+ return 0;
+}
+
+static int
+test_member_perf(void)
+{
+
+ if (run_all_tbl_perf_tests() < 0)
+ return -1;
+
+ return 0;
+}
+
+REGISTER_TEST_COMMAND(member_perf_autotest, test_member_perf);
--
2.7.4
Yipeng Wang
2017-09-02 01:24:41 UTC
Permalink
This patch adds the documentation for membership library.

Signed-off-by: Yipeng Wang <***@intel.com>
---
doc/api/doxy-api-index.md | 3 +-
doc/api/doxy-api.conf | 1 +
doc/guides/prog_guide/img/member_i1.svg | 1613 +++++++++++++++++++++++++++++++
doc/guides/prog_guide/img/member_i2.svg | 36 +
doc/guides/prog_guide/img/member_i3.svg | 148 +++
doc/guides/prog_guide/img/member_i4.svg | 450 +++++++++
doc/guides/prog_guide/img/member_i5.svg | 163 ++++
doc/guides/prog_guide/img/member_i6.svg | 332 +++++++
doc/guides/prog_guide/img/member_i7.svg | 399 ++++++++
doc/guides/prog_guide/index.rst | 14 +
doc/guides/prog_guide/member_lib.rst | 440 +++++++++
doc/guides/rel_notes/release_17_11.rst | 17 +
12 files changed, 3615 insertions(+), 1 deletion(-)
create mode 100644 doc/guides/prog_guide/img/member_i1.svg
create mode 100644 doc/guides/prog_guide/img/member_i2.svg
create mode 100644 doc/guides/prog_guide/img/member_i3.svg
create mode 100644 doc/guides/prog_guide/img/member_i4.svg
create mode 100644 doc/guides/prog_guide/img/member_i5.svg
create mode 100644 doc/guides/prog_guide/img/member_i6.svg
create mode 100644 doc/guides/prog_guide/img/member_i7.svg
create mode 100644 doc/guides/prog_guide/member_lib.rst

diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 19e0d4f..fe87e09 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -105,7 +105,8 @@ The public API headers are grouped by topics:
[LPM IPv4 route] (@ref rte_lpm.h),
[LPM IPv6 route] (@ref rte_lpm6.h),
[ACL] (@ref rte_acl.h),
- [EFD] (@ref rte_efd.h)
+ [EFD] (@ref rte_efd.h),
+ [member] (@ref rte_member.h)

- **QoS**:
[metering] (@ref rte_meter.h),
diff --git a/doc/api/doxy-api.conf b/doc/api/doxy-api.conf
index 823554f..b792d6d 100644
--- a/doc/api/doxy-api.conf
+++ b/doc/api/doxy-api.conf
@@ -58,6 +58,7 @@ INPUT = doc/api/doxy-api-index.md \
lib/librte_mempool \
lib/librte_meter \
lib/librte_metrics \
+ lib/librte_member \
lib/librte_net \
lib/librte_pdump \
lib/librte_pipeline \
diff --git a/doc/guides/prog_guide/img/member_i1.svg b/doc/guides/prog_guide/img/member_i1.svg
new file mode 100644
index 0000000..fc5f56a
--- /dev/null
+++ b/doc/guides/prog_guide/img/member_i1.svg
@@ -0,0 +1,1613 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.0//EN" "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd">
+<!-- Generated by Microsoft Visio 11.0, SVG Export, v1.0 memship_i1.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+ xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="7.18709in" height="4.75757in"
+ viewBox="0 0 517.471 342.545" xml:space="preserve" color-interpolation-filters="sRGB" class="st61">
+ <v:documentProperties v:langID="1033" v:viewMarkup="false">
+ <v:userDefs>
+ <v:ud v:nameU="msvSubprocessMaster" v:prompt="" v:val="VT4(Rectangle)"/>
+ <v:ud v:nameU="msvNoAutoConnect" v:val="VT0(1):26"/>
+ </v:userDefs>
+ </v:documentProperties>
+
+ <style type="text/css">
+ <![CDATA[
+ .st1 {visibility:visible}
+ .st2 {fill:none;stroke:#5b9bd5;stroke-opacity:0.22;stroke-width:3}
+ .st3 {fill:#5b9bd5;font-family:Calibri;font-size:0.666664em;opacity:0.219608}
+ .st4 {font-size:1em}
+ .st5 {fill:none;stroke:#41719c;stroke-width:3}
+ .st6 {fill:#5b9bd5;font-family:Calibri;font-size:0.666664em}
+ .st7 {fill:#5b9bd5;font-family:Calibri;font-size:0.75em;opacity:0.219608}
+ .st8 {fill:#5b9bd5;font-family:Calibri;font-size:0.75em}
+ .st9 {fill:#5b9bd5;fill-opacity:0.22;stroke:#5b9bd5;stroke-opacity:0.22;stroke-width:0.25}
+ .st10 {fill:#5b9bd5;stroke:#c8c8c8;stroke-width:0.25}
+ .st11 {fill:none;stroke:none;stroke-width:0.25}
+ .st12 {fill:#ffffff;font-family:Calibri;font-size:0.499992em;font-weight:bold}
+ .st13 {fill:#ffffff;font-family:Calibri;font-size:0.75em;font-weight:bold}
+ .st14 {marker-end:url(#mrkr5-63);stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st15 {fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-opacity:1;stroke-width:0.28409090909091}
+ .st16 {fill:#5b9bd5;font-family:Calibri;font-size:0.499992em;font-weight:bold}
+ .st17 {stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st18 {fill:#feffff;font-family:Calibri;font-size:0.499992em}
+ .st19 {fill:#deebf6;stroke:#c8c8c8;stroke-width:0.25}
+ .st20 {fill:#000000;font-family:Calibri;font-size:0.499992em}
+ .st21 {marker-end:url(#mrkr5-178);stroke:#ff0000;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st22 {fill:#ff0000;fill-opacity:1;stroke:#ff0000;stroke-opacity:1;stroke-width:0.28409090909091}
+ .st23 {fill:#ff0000;font-family:Calibri;font-size:0.666664em}
+ .st24 {fill:#5b9bd5;fill-opacity:0.22}
+ .st25 {stroke:#5b9bd5;stroke-opacity:0.22;stroke-width:0.25}
+ .st26 {fill:#ffffff}
+ .st27 {stroke:#0070c0;stroke-width:0.25}
+ .st28 {fill:#5b9bd5;stroke:#0070c0;stroke-width:0.25}
+ .st29 {fill:#5b9bd5;stroke:#ffffff;stroke-width:0.25}
+ .st30 {fill:#5b9bd5}
+ .st31 {stroke:#c8c8c8;stroke-width:0.25}
+ .st32 {fill:#acccea;stroke:#c8c8c8;stroke-width:0.25}
+ .st33 {fill:#5b9bd5;fill-opacity:0.22;stroke:none;stroke-linecap:butt;stroke-width:0.75}
+ .st34 {fill:#000000;fill-opacity:0;stroke:none;stroke-linecap:butt;stroke-width:0.75}
+ .st35 {fill:url(#grad30-309);stroke:none;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.75}
+ .st36 {fill:url(#grad25-313);stroke:none;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.75}
+ .st37 {fill:url(#grad35-317);stroke:#308dda;stroke-linecap:butt;stroke-width:0.130208}
+ .st38 {fill:url(#grad36-325);stroke:none;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.75}
+ .st39 {fill:url(#grad40-335);stroke:#000000;stroke-linecap:butt;stroke-width:0.130208}
+ .st40 {fill:url(#grad39-342);stroke:none;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.75}
+ .st41 {fill:url(#grad40-355);stroke:none;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.75}
+ .st42 {fill:none}
+ .st43 {stroke:#308dda;stroke-linecap:butt;stroke-width:0.130208}
+ .st44 {stroke:#ffffff;stroke-linecap:butt;stroke-width:0.130208}
+ .st45 {fill:url(#grad30-383);stroke:none;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.75}
+ .st46 {fill:url(#grad36-396);stroke:none;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.75}
+ .st47 {fill:none;stroke:#c8c8c8;stroke-width:0.75}
+ .st48 {fill:#9a9a9a;stroke:#000000;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.0833333}
+ .st49 {fill:url(#grad40-415);stroke:#000000;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.0833333}
+ .st50 {fill:url(#grad40-419);stroke:#000000;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.0833333}
+ .st51 {stroke:#000000;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.25}
+ .st52 {fill:url(#grad35-430);stroke:#000000;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.0833333}
+ .st53 {stroke:#c8c8c8;stroke-width:0.75}
+ .st54 {stroke:#4f88bb;stroke-width:0.75}
+ .st55 {fill:#feffff;font-family:Calibri;font-size:0.416656em}
+ .st56 {fill:#5b9bd5;fill-opacity:0.25;stroke:#5b9bd5;stroke-opacity:0.25;stroke-width:0.75}
+ .st57 {fill:#4f88bb;stroke:#41719c;stroke-width:0.75}
+ .st58 {fill:none;stroke:#5b9bd5;stroke-opacity:0.22;stroke-width:2.25}
+ .st59 {fill:none;stroke:#0070c0;stroke-width:2.25}
+ .st60 {fill:#595959;font-family:Arial;font-size:0.666664em}
+ .st61 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+ ]]>
+ </style>
+
+ <defs id="Patterns_And_Gradients">
+ <linearGradient id="grad30-309" v:fillPattern="30" v:foreground="#97c2e6" v:background="#4274a2" x1="0" y1="1" x2="0"
+ y2="0">
+ <stop offset="0" style="stop-color:#97c2e6;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#4274a2;stop-opacity:1"/>
+ </linearGradient>
+ <linearGradient id="grad25-313" v:fillPattern="25" v:foreground="#5491d3" v:background="#246ba6" x1="0" y1="0" x2="1"
+ y2="0">
+ <stop offset="0" style="stop-color:#5491d3;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#246ba6;stop-opacity:1"/>
+ </linearGradient>
+ <pattern id="grad35-317" v:fillPattern="35" v:foreground="#569bd3" v:background="#aed0ec" x="0" y="0" width="1" height="1"
+ patternContentUnits="objectBoundingBox">
+ <path d="M 0.5 0.5 L 0 0 L 0 1 z" style="fill:url(#grad27-318)"/>
+ <path d="M 0.5 0.5 L 1 0 L 1 1 z" style="fill:url(#grad25-319)"/>
+ <path d="M 0.5 0.5 L 0 0 L 1 0 z" style="fill:url(#grad30-320)"/>
+ <path d="M 0.5 0.5 L 0 1 L 1 1 z" style="fill:url(#grad28-321)"/>
+ </pattern>
+ <linearGradient id="grad27-318" v:fillPattern="35" v:foreground="#569bd3" v:background="#aed0ec" x1="1" y1="0" x2="0"
+ y2="0">
+ <stop offset="0" style="stop-color:#569bd3;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#aed0ec;stop-opacity:1"/>
+ </linearGradient>
+ <linearGradient id="grad25-319" v:fillPattern="35" v:foreground="#569bd3" v:background="#aed0ec" x1="0" y1="0" x2="1"
+ y2="0">
+ <stop offset="0" style="stop-color:#569bd3;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#aed0ec;stop-opacity:1"/>
+ </linearGradient>
+ <linearGradient id="grad30-320" v:fillPattern="35" v:foreground="#569bd3" v:background="#aed0ec" x1="0" y1="1" x2="0"
+ y2="0">
+ <stop offset="0" style="stop-color:#569bd3;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#aed0ec;stop-opacity:1"/>
+ </linearGradient>
+ <linearGradient id="grad28-321" v:fillPattern="35" v:foreground="#569bd3" v:background="#aed0ec" x1="0" y1="0" x2="0"
+ y2="1">
+ <stop offset="0" style="stop-color:#569bd3;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#aed0ec;stop-opacity:1"/>
+ </linearGradient>
+ <radialGradient id="grad36-325" v:fillPattern="36" v:foreground="#c0dff1" v:background="#246ba6" cx="0" cy="0" r="1">
+ <stop offset="0" style="stop-color:#c0dff1;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#246ba6;stop-opacity:1"/>
+ </radialGradient>
+ <radialGradient id="grad40-335" v:fillPattern="40" v:foreground="#c8e5c8" v:background="#19bf19" cx="0.5" cy="0.5" r="0.5">
+ <stop offset="0" style="stop-color:#c8e5c8;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#19bf19;stop-opacity:1"/>
+ </radialGradient>
+ <radialGradient id="grad39-342" v:fillPattern="39" v:foreground="#5599d7" v:background="#b9daf2" cx="1" cy="1" r="1">
+ <stop offset="0" style="stop-color:#5599d7;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#b9daf2;stop-opacity:1"/>
+ </radialGradient>
+ <radialGradient id="grad40-355" v:fillPattern="40" v:foreground="#5599d7" v:background="#214383" cx="0.5" cy="0.5" r="0.5">
+ <stop offset="0" style="stop-color:#5599d7;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#214383;stop-opacity:1"/>
+ </radialGradient>
+ <linearGradient id="grad30-383" v:fillPattern="30" v:foreground="#97c2e6" v:background="#6ba4dc" x1="0" y1="1" x2="0"
+ y2="0">
+ <stop offset="0" style="stop-color:#97c2e6;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#6ba4dc;stop-opacity:1"/>
+ </linearGradient>
+ <radialGradient id="grad36-396" v:fillPattern="36" v:foreground="#89bee9" v:background="#b9daf2" cx="0" cy="0" r="1">
+ <stop offset="0" style="stop-color:#89bee9;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#b9daf2;stop-opacity:1"/>
+ </radialGradient>
+ <radialGradient id="grad40-415" v:fillPattern="40" v:foreground="#000000" v:background="#ffffff" cx="0.5" cy="0.5" r="0.5">
+ <stop offset="0" style="stop-color:#000000;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#ffffff;stop-opacity:1"/>
+ </radialGradient>
+ <radialGradient id="grad40-419" v:fillPattern="40" v:foreground="#ffffff" v:background="#9a9a9a" cx="0.5" cy="0.5" r="0.5">
+ <stop offset="0" style="stop-color:#ffffff;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#9a9a9a;stop-opacity:1"/>
+ </radialGradient>
+ <pattern id="grad35-430" v:fillPattern="35" v:foreground="#ffffff" v:background="#ffcc00" x="0" y="0" width="1" height="1"
+ patternContentUnits="objectBoundingBox">
+ <path d="M 0.5 0.5 L 0 0 L 0 1 z" style="fill:url(#grad27-431)"/>
+ <path d="M 0.5 0.5 L 1 0 L 1 1 z" style="fill:url(#grad25-432)"/>
+ <path d="M 0.5 0.5 L 0 0 L 1 0 z" style="fill:url(#grad30-433)"/>
+ <path d="M 0.5 0.5 L 0 1 L 1 1 z" style="fill:url(#grad28-434)"/>
+ </pattern>
+ <linearGradient id="grad27-431" v:fillPattern="35" v:foreground="#ffffff" v:background="#ffcc00" x1="1" y1="0" x2="0"
+ y2="0">
+ <stop offset="0" style="stop-color:#ffffff;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#ffcc00;stop-opacity:1"/>
+ </linearGradient>
+ <linearGradient id="grad25-432" v:fillPattern="35" v:foreground="#ffffff" v:background="#ffcc00" x1="0" y1="0" x2="1"
+ y2="0">
+ <stop offset="0" style="stop-color:#ffffff;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#ffcc00;stop-opacity:1"/>
+ </linearGradient>
+ <linearGradient id="grad30-433" v:fillPattern="35" v:foreground="#ffffff" v:background="#ffcc00" x1="0" y1="1" x2="0"
+ y2="0">
+ <stop offset="0" style="stop-color:#ffffff;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#ffcc00;stop-opacity:1"/>
+ </linearGradient>
+ <linearGradient id="grad28-434" v:fillPattern="35" v:foreground="#ffffff" v:background="#ffcc00" x1="0" y1="0" x2="0"
+ y2="1">
+ <stop offset="0" style="stop-color:#ffffff;stop-opacity:1"/>
+ <stop offset="1" style="stop-color:#ffcc00;stop-opacity:1"/>
+ </linearGradient>
+ </defs>
+ <defs id="Markers">
+ <g id="lend5">
+ <path d="M 2 1 L 0 0 L 1.98117 -0.993387 C 1.67173 -0.364515 1.67301 0.372641 1.98465 1.00043 " style="stroke:none"/>
+ </g>
+ <marker id="mrkr5-63" class="st15" v:arrowType="5" v:arrowSize="2" v:setback="6.16" refX="-6.16" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-3.52,-3.52) "/>
+ </marker>
+ <marker id="mrkr5-178" class="st22" v:arrowType="5" v:arrowSize="2" v:setback="5.8" refX="-5.8" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-3.52,-3.52) "/>
+ </marker>
+ </defs>
+ <g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeOrder" v:val="VT0(0):26"/>
+ </v:userDefs>
+ <title>Page-1</title>
+ <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+ <v:layer v:name="Flowchart" v:index="0"/>
+ <g id="group165-1" transform="translate(21.7794,-24.0978)" v:mID="165" v:groupContext="group">
+ <title>Sheet.165</title>
+ <g id="group1-2" transform="translate(308.647,-25.7109)" v:mID="1" v:groupContext="group">
+ <title>Sheet.1</title>
+ <g id="shape2-3" v:mID="2" v:groupContext="shape" transform="translate(11.5732,-58.1913)">
+ <title>Circle</title>
+ <desc>List 1 matching Criteria 1</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="27.2233" cy="315.322" width="47.65" height="40.835"/>
+ <g id="shadow2-4" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 315.32 A27.2233 27.2233 0 1 1 54.45 315.32 A27.2233 27.2233 0 1 1 0 315.32 Z" class="st2"/>
+ <text x="18.79" y="308.12" class="st3" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>List 1 <tspan
+ x="12.08" dy="1.2em" class="st4">matching </tspan><tspan x="12.29" dy="1.2em" class="st4">Criteria </tspan>1</text> </g>
+ <path d="M0 315.32 A27.2233 27.2233 0 1 1 54.45 315.32 A27.2233 27.2233 0 1 1 0 315.32 Z" class="st5"/>
+ <text x="18.79" y="308.12" class="st6" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>List 1 <tspan
+ x="12.08" dy="1.2em" class="st4">matching </tspan><tspan x="12.29" dy="1.2em" class="st4">Criteria </tspan>1</text> </g>
+ <g id="shape3-13" v:mID="3" v:groupContext="shape" transform="translate(58.9839,-58.9839)">
+ <title>Circle.23</title>
+ <desc>List 2</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="27.2233" cy="315.322" width="47.65" height="40.835"/>
+ <g id="shadow3-14" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 315.32 A27.2233 27.2233 0 1 1 54.45 315.32 A27.2233 27.2233 0 1 1 0 315.32 Z" class="st2"/>
+ <text x="17.73" y="318.02" class="st7" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>List 2</text> </g>
+ <path d="M0 315.32 A27.2233 27.2233 0 1 1 54.45 315.32 A27.2233 27.2233 0 1 1 0 315.32 Z" class="st5"/>
+ <text x="17.73" y="318.02" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>List 2</text> </g>
+ <g id="shape4-19" v:mID="4" v:groupContext="shape">
+ <title>Circle.24</title>
+ <desc>List 1 matching Criteria 1</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="27.2233" cy="315.322" width="47.65" height="40.835"/>
+ <g id="shadow4-20" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 315.32 A27.2233 27.2233 0 1 1 54.45 315.32 A27.2233 27.2233 0 1 1 0 315.32 Z" class="st2"/>
+ <text x="18.79" y="308.12" class="st3" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>List 1 <tspan
+ x="12.08" dy="1.2em" class="st4">matching </tspan><tspan x="12.29" dy="1.2em" class="st4">Criteria </tspan>1</text> </g>
+ <path d="M0 315.32 A27.2233 27.2233 0 1 1 54.45 315.32 A27.2233 27.2233 0 1 1 0 315.32 Z" class="st5"/>
+ <text x="18.79" y="308.12" class="st6" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>List 1 <tspan
+ x="12.08" dy="1.2em" class="st4">matching </tspan><tspan x="12.29" dy="1.2em" class="st4">Criteria </tspan>1</text> </g>
+ <g id="group5-29" transform="translate(50.7413,-4.53722)" v:mID="5" v:groupContext="group">
+ <title>Sheet.5</title>
+ <g id="shape6-30" v:mID="6" v:groupContext="shape" transform="translate(344.2,300.5) rotate(90)">
+ <title>Triangle</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow6-31" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,1.9728,-0.3456)" class="st1">
+ <path d="M42.04 342.55 L21.02 318.64 L0 342.55 L42.04 342.55 Z" class="st9"/>
+ </g>
+ <path d="M42.04 342.55 L21.02 318.64 L0 342.55 L42.04 342.55 Z" class="st10"/>
+ </g>
+ <g id="shape7-34" v:mID="7" v:groupContext="shape" transform="translate(-0.884982,-14.7157)">
+ <title>Sheet.7</title>
+ <desc>setsum</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="12.9268" cy="336.238" width="25.86" height="12.6135"/>
+ <rect x="0" y="329.932" width="25.8535" height="12.6135" class="st11"/>
+ <text x="6.37" y="334.44" class="st12" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>setsu<tspan
+ x="10.49" dy="1.2em" class="st4">m</tspan></text> </g>
+ </g>
+ <g id="shape8-38" v:mID="8" v:groupContext="shape" transform="translate(72.5955,0)">
+ <title>Circle.29</title>
+ <desc>List 2 matching Criteria 2</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="27.2233" cy="315.322" width="47.65" height="40.835"/>
+ <g id="shadow8-39" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 315.32 A27.2233 27.2233 0 1 1 54.45 315.32 A27.2233 27.2233 0 1 1 0 315.32 Z" class="st2"/>
+ <text x="18.79" y="308.12" class="st3" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>List 2 <tspan
+ x="12.08" dy="1.2em" class="st4">matching </tspan><tspan x="12.29" dy="1.2em" class="st4">Criteria </tspan>2</text> </g>
+ <path d="M0 315.32 A27.2233 27.2233 0 1 1 54.45 315.32 A27.2233 27.2233 0 1 1 0 315.32 Z" class="st5"/>
+ <text x="18.79" y="308.12" class="st6" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>List 2 <tspan
+ x="12.08" dy="1.2em" class="st4">matching </tspan><tspan x="12.29" dy="1.2em" class="st4">Criteria </tspan>2</text> </g>
+ </g>
+ <g id="group9-48" transform="translate(31.6515,-49.9094)" v:mID="9" v:groupContext="group">
+ <title>Sheet.9</title>
+ <g id="group10-49" transform="translate(99.5691,0)" v:mID="10" v:groupContext="group">
+ <title>Sheet.10</title>
+ <g id="shape11-50" v:mID="11" v:groupContext="shape" transform="translate(346.175,275.999) rotate(90)">
+ <title>Triangle</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow11-51" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,1.9728,-0.3456)" class="st1">
+ <path d="M66.55 342.55 L33.27 290.12 L0 342.55 L66.55 342.55 Z" class="st9"/>
+ </g>
+ <path d="M66.55 342.55 L33.27 290.12 L0 342.55 L66.55 342.55 Z" class="st10"/>
+ </g>
+ <g id="shape12-54" v:mID="12" v:groupContext="shape" transform="translate(355.063,285.074) rotate(90)">
+ <title>Sheet.12</title>
+ <desc>Set Summary</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="24.1985" cy="332.563" width="48.4" height="19.9638"/>
+ <rect x="0" y="322.581" width="48.397" height="19.9638" class="st11"/>
+ <text x="18.25" y="329.86" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Set <tspan
+ x="6.38" dy="1.2em" class="st4">Summary</tspan></text> </g>
+ </g>
+ <g id="shape13-58" v:mID="13" v:groupContext="shape" transform="translate(57.5835,-54.4467)">
+ <title>Sheet.13</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L38.9 342.55" class="st14"/>
+ </g>
+ <g id="shape14-64" v:mID="14" v:groupContext="shape" transform="translate(20.2363,-51.8439)">
+ <title>Sheet.14</title>
+ <desc>Flow Key</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="25.3328" cy="333.471" width="50.67" height="18.1489"/>
+ <rect x="0" y="324.396" width="50.6656" height="18.1489" class="st11"/>
+ <text x="14.12" y="335.27" class="st16" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow Key</text> </g>
+ <g id="shape15-67" v:mID="15" v:groupContext="shape" transform="translate(5.02911,1.60865) rotate(-26.0815)">
+ <title>Sheet.15</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L39.25 342.55" class="st14"/>
+ </g>
+ <g id="shape16-72" v:mID="16" v:groupContext="shape" transform="translate(155.629,-33.273)">
+ <title>Sheet.16</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L38.34 342.55" class="st14"/>
+ </g>
+ <g id="shape17-77" v:mID="17" v:groupContext="shape" transform="translate(304.141,0.595416) rotate(25.6934)">
+ <title>Sheet.17</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L42.68 342.55" class="st14"/>
+ </g>
+ <g id="shape18-82" v:mID="18" v:groupContext="shape" transform="translate(102.642,654.842) rotate(180)">
+ <title>Sheet.18</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L30.14 342.55" class="st14"/>
+ </g>
+ <g id="shape19-87" v:mID="19" v:groupContext="shape" transform="translate(-15.1809,-33.9928)">
+ <title>Sheet.19</title>
+ <desc>New Flow =&#62; New Assignment</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="42.75" cy="338.045" width="85.5" height="9"/>
+ <rect x="0" y="333.545" width="85.5" height="9" class="st11"/>
+ <text x="5.06" y="339.85" class="st16" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>New Flow =&#62; New Assignment</text> </g>
+ <g id="shape20-90" v:mID="20" v:groupContext="shape" transform="translate(102.844,679.041) rotate(180)">
+ <title>Sheet.20</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L30.14 342.55" class="st14"/>
+ </g>
+ <g id="shape21-95" v:mID="21" v:groupContext="shape" transform="translate(-35.4309,-11.4928)">
+ <title>Sheet.21</title>
+ <desc>Old Flow =&#62; forward to specific thread</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="337.971" width="108" height="9.14889"/>
+ <rect x="0" y="333.396" width="108" height="9.14889" class="st11"/>
+ <text x="6.36" y="339.77" class="st16" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Old Flow =&#62; forward to specific thread</text> </g>
+ <g id="shape22-98" v:mID="22" v:groupContext="shape" transform="translate(541.496,275.999) rotate(90)">
+ <title>Sheet.22</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 335.81 C2.14 344.21 5.09 343.6 7.56 340.31 C10.62 336.25 12.94 328.1 18.15 335.81" class="st17"/>
+ </g>
+ <g id="shape23-101" v:mID="23" v:groupContext="shape" transform="translate(541.496,300.198) rotate(90)">
+ <title>Sheet.23</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 335.81 C2.14 344.21 5.09 343.6 7.56 340.31 C10.62 336.25 12.94 328.1 18.15 335.81" class="st17"/>
+ </g>
+ <g id="shape24-104" v:mID="24" v:groupContext="shape" transform="translate(541.496,324.396) rotate(90)">
+ <title>Sheet.24</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 335.81 C2.14 344.21 5.09 343.6 7.56 340.31 C10.62 336.25 12.94 328.1 18.15 335.81" class="st17"/>
+ </g>
+ </g>
+ <g id="group25-107" transform="translate(285.961,-178.628)" v:mID="25" v:groupContext="group">
+ <title>Sheet.25</title>
+ <g id="shape26-108" v:mID="26" v:groupContext="shape" transform="translate(51.2583,-51.2583)">
+ <title>Circle</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow26-109" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st9"/>
+ </g>
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st10"/>
+ </g>
+ <g id="shape27-112" v:mID="27" v:groupContext="shape" transform="translate(107.177,-55.9182)">
+ <title>Circle.156</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow27-113" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st9"/>
+ </g>
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st10"/>
+ </g>
+ <g id="shape28-116" v:mID="28" v:groupContext="shape" transform="translate(79.2174,-83.8773)">
+ <title>Circle.157</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow28-117" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st9"/>
+ </g>
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st10"/>
+ </g>
+ <g id="shape29-120" v:mID="29" v:groupContext="shape" transform="translate(153.775,-51.2583)">
+ <title>Circle.158</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow29-121" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st9"/>
+ </g>
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st10"/>
+ </g>
+ <g id="shape30-124" v:mID="30" v:groupContext="shape" transform="translate(93.197,-18.6394)">
+ <title>Circle.159</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow30-125" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st9"/>
+ </g>
+ <path d="M0 333.23 A9.3197 9.3197 0 0 1 18.64 333.23 A9.3197 9.3197 0 0 1 0 333.23 Z" class="st10"/>
+ </g>
+ <g id="shape31-128" v:mID="31" v:groupContext="shape" transform="translate(27.4102,-57.9329) rotate(-7.12502)">
+ <title>Sheet.31</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L31.41 342.55" class="st14"/>
+ </g>
+ <g id="shape32-133" v:mID="32" v:groupContext="shape" transform="translate(182.13,-60.5772) rotate(9.46232)">
+ <title>Sheet.32</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L22.18 342.55" class="st14"/>
+ </g>
+ <g id="shape33-138" v:mID="33" v:groupContext="shape" transform="translate(47.8843,595.237) rotate(-160.346)">
+ <title>Sheet.33</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L63.11 342.55" class="st14"/>
+ </g>
+ <g id="shape34-143" v:mID="34" v:groupContext="shape" transform="translate(292.945,525.785) rotate(141.977)">
+ <title>Sheet.34</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L20.97 342.55" class="st14"/>
+ </g>
+ <g id="shape35-148" v:mID="35" v:groupContext="shape" transform="translate(-95.8971,591.793) rotate(-145.945)">
+ <title>Sheet.35</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L28.55 342.55" class="st14"/>
+ </g>
+ <g id="shape36-153" v:mID="36" v:groupContext="shape" transform="translate(37.2788,2.27374E-013)">
+ <title>Rectangle.167</title>
+ <desc>SUM</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="10.8652" cy="335.555" width="21.74" height="13.9795"/>
+ <g id="shadow36-154" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="328.566" width="21.7305" height="13.9795" class="st9"/>
+ </g>
+ <rect x="0" y="328.566" width="21.7305" height="13.9795" class="st10"/>
+ <text x="5" y="337.36" class="st18" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>SUM</text> </g>
+ <g id="shape37-158" v:mID="37" v:groupContext="shape" transform="translate(55.9182,2.27374E-013)">
+ <title>Rectangle.168</title>
+ <desc>Packet</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="23.2992" cy="335.555" width="46.6" height="13.9795"/>
+ <g id="shadow37-159" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="328.566" width="46.5985" height="13.9795" class="st9"/>
+ </g>
+ <rect x="0" y="328.566" width="46.5985" height="13.9795" class="st19"/>
+ <text x="15.18" y="337.36" class="st20" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Packet</text> </g>
+ <g id="shape38-163" v:mID="38" v:groupContext="shape" transform="translate(-1.65867E-013,-32.6189)">
+ <title>Rectangle.169</title>
+ <desc>SUM</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="10.3796" cy="335.555" width="20.76" height="13.9795"/>
+ <g id="shadow38-164" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="328.566" width="20.7593" height="13.9795" class="st9"/>
+ </g>
+ <rect x="0" y="328.566" width="20.7593" height="13.9795" class="st10"/>
+ <text x="4.51" y="337.36" class="st18" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>SUM</text> </g>
+ <g id="shape39-168" v:mID="39" v:groupContext="shape" transform="translate(18.6394,-32.6189)">
+ <title>Rectangle.170</title>
+ <desc>Packet</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="23.2992" cy="335.555" width="46.6" height="13.9795"/>
+ <g id="shadow39-169" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="328.566" width="46.5985" height="13.9795" class="st9"/>
+ </g>
+ <rect x="0" y="328.566" width="46.5985" height="13.9795" class="st19"/>
+ <text x="15.18" y="337.36" class="st20" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Packet</text> </g>
+ <g id="shape40-173" v:mID="40" v:groupContext="shape" transform="translate(197.019,626.053) rotate(161.565)">
+ <title>Sheet.40</title>
+ <path d="M0 328.31 A55.7483 27.2427 -124.2 0 0 42.37 334.19 L42.47 333.85" class="st21"/>
+ </g>
+ <g id="shape41-179" v:mID="41" v:groupContext="shape" transform="translate(154.607,584.177) rotate(161.121)">
+ <title>Sheet.41</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 319.39 A80.5593 29.9756 -101.99 0 0 41.7 325.37 L41.79 325.02" class="st21"/>
+ </g>
+ <g id="shape42-184" v:mID="42" v:groupContext="shape" transform="translate(3.02481,-66.7025)">
+ <title>Sheet.42</title>
+ <desc>Encode ID</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="19.4569" cy="335.555" width="38.92" height="13.9795"/>
+ <rect x="0" y="328.566" width="38.9138" height="13.9795" class="st11"/>
+ <text x="7.51" y="333.16" class="st23" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Encode <tspan
+ x="15.99" dy="1.2em" class="st4">ID</tspan></text> </g>
+ </g>
+ <g id="group43-188" transform="translate(12.0993,-165.858)" v:mID="43" v:groupContext="group">
+ <title>Sheet.43</title>
+ <g id="group44-189" transform="translate(7.21495,-75.757)" v:mID="44" v:groupContext="group" v:layerMember="0">
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ <v:ud v:nameU="ConnGap" v:prompt="" v:val="VT0(0.083333333333333):0"/>
+ </v:userDefs>
+ <title>User</title>
+ <g id="shape45-190" v:mID="45" v:groupContext="shape" v:layerMember="0"
+ transform="translate(13.3353,-1.13687E-013)">
+ <title>Sheet.45</title>
+ <g id="shadow45-191" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42 ZM20.96
+ 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52
+ 13.66 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55
+ L21.12 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36
+ 329.46 C22.2 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77 Z"
+ class="st24"/>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42"
+ class="st25"/>
+ <path d="M20.96 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52
+ 13.66 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55
+ L21.12 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36
+ 329.46 C22.2 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77"
+ class="st25"/>
+ <path d="M18.1 342.55 L18.1 338.37" class="st25"/>
+ <path d="M15.03 337.32 L15.03 333.71" class="st25"/>
+ <path d="M21.12 337.32 L21.12 333.71" class="st25"/>
+ <path d="M0 337.32 L13.47 337.32" class="st25"/>
+ </g>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42 ZM20.96
+ 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52 13.66
+ 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55 L21.12
+ 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36 329.46 C22.2
+ 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77 Z" class="st26"/>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42"
+ class="st27"/>
+ <path d="M20.96 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52 13.66
+ 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55 L21.12
+ 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36 329.46 C22.2
+ 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77" class="st27"/>
+ <path d="M18.1 342.55 L18.1 338.37" class="st27"/>
+ <path d="M15.03 337.32 L15.03 333.71" class="st27"/>
+ <path d="M21.12 337.32 L21.12 333.71" class="st27"/>
+ <path d="M0 337.32 L13.47 337.32" class="st27"/>
+ </g>
+ <g id="shape46-206" v:mID="46" v:groupContext="shape" v:layerMember="0" transform="translate(0,-8.39743)">
+ <title>Sheet.46</title>
+ <g id="shadow46-207" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M21.09 325.52 C21.09 325.13 20.96 324.79 20.74 324.51 C20.59 324.32 20.4 324.16 20.19 324.06
+ C19.97 323.95 19.72 323.89 19.46 323.89 L3.55 323.89 C3.16 323.89 2.82 324.02 2.54 324.23
+ C2.35 324.38 2.19 324.57 2.09 324.79 C1.98 325.01 1.92 325.25 1.92 325.52 L1.92 336.04 L21.09
+ 336.04 L21.09 325.52 ZM21.18 337.33 L1.77 337.33 L0 340.51 L0 342.55 L23.06 342.55 L23.06
+ 340.51 L21.18 337.33 Z" class="st9"/>
+ </g>
+ <path d="M21.09 325.52 C21.09 325.13 20.96 324.79 20.74 324.51 C20.59 324.32 20.4 324.16 20.19 324.06 C19.97
+ 323.95 19.72 323.89 19.46 323.89 L3.55 323.89 C3.16 323.89 2.82 324.02 2.54 324.23 C2.35 324.38
+ 2.19 324.57 2.09 324.79 C1.98 325.01 1.92 325.25 1.92 325.52 L1.92 336.04 L21.09 336.04 L21.09
+ 325.52 ZM21.18 337.33 L1.77 337.33 L0 340.51 L0 342.55 L23.06 342.55 L23.06 340.51 L21.18 337.33
+ Z" class="st28"/>
+ </g>
+ <g id="shape47-210" v:mID="47" v:groupContext="shape" v:layerMember="0" transform="translate(3.19243,-16.175)">
+ <title>Sheet.47</title>
+ <v:userDefs>
+ <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(#5b9bd5)"/>
+ </v:userDefs>
+ <path d="M16.62 342.55 L16.62 333.29 C16.62 333.23 16.61 333.18 16.58 333.13 C16.55 333.07 16.5 333.02 16.44
+ 332.98 C16.39 332.95 16.33 332.94 16.27 332.94 L0.35 332.94 C0.29 332.94 0.24 332.95 0.19 332.98
+ C0.13 333.01 0.08 333.07 0.04 333.12 C0.02 333.17 0 333.23 0 333.29 L0 342.55 L16.62 342.55
+ Z" class="st29"/>
+ </g>
+ <g id="shape48-212" v:mID="48" v:groupContext="shape" v:layerMember="0" transform="translate(1.97942,-10.81)">
+ <title>Sheet.48</title>
+ <v:userDefs>
+ <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(#5b9bd5)"/>
+ </v:userDefs>
+ <path d="M0.96 340.83 L0 342.55 L19.06 342.55 L18.1 340.83 L0.96 340.83 Z" class="st26"/>
+ </g>
+ </g>
+ <g id="group49-215" transform="translate(7.21495,-47.1858)" v:mID="49" v:groupContext="group" v:layerMember="0">
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ <v:ud v:nameU="ConnGap" v:prompt="" v:val="VT0(0.083333333333333):0"/>
+ </v:userDefs>
+ <title>User.7</title>
+ <g id="shape50-216" v:mID="50" v:groupContext="shape" v:layerMember="0"
+ transform="translate(13.3353,-1.13687E-013)">
+ <title>Sheet.50</title>
+ <g id="shadow50-217" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42 ZM20.96
+ 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52
+ 13.66 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55
+ L21.12 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36
+ 329.46 C22.2 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77 Z"
+ class="st24"/>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42"
+ class="st25"/>
+ <path d="M20.96 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52
+ 13.66 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55
+ L21.12 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36
+ 329.46 C22.2 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77"
+ class="st25"/>
+ <path d="M18.1 342.55 L18.1 338.37" class="st25"/>
+ <path d="M15.03 337.32 L15.03 333.71" class="st25"/>
+ <path d="M21.12 337.32 L21.12 333.71" class="st25"/>
+ <path d="M0 337.32 L13.47 337.32" class="st25"/>
+ </g>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42 ZM20.96
+ 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52 13.66
+ 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55 L21.12
+ 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36 329.46 C22.2
+ 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77 Z" class="st26"/>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42"
+ class="st27"/>
+ <path d="M20.96 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52 13.66
+ 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55 L21.12
+ 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36 329.46 C22.2
+ 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77" class="st27"/>
+ <path d="M18.1 342.55 L18.1 338.37" class="st27"/>
+ <path d="M15.03 337.32 L15.03 333.71" class="st27"/>
+ <path d="M21.12 337.32 L21.12 333.71" class="st27"/>
+ <path d="M0 337.32 L13.47 337.32" class="st27"/>
+ </g>
+ <g id="shape51-232" v:mID="51" v:groupContext="shape" v:layerMember="0" transform="translate(0,-8.39743)">
+ <title>Sheet.51</title>
+ <g id="shadow51-233" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M21.09 325.52 C21.09 325.13 20.96 324.79 20.74 324.51 C20.59 324.32 20.4 324.16 20.19 324.06
+ C19.97 323.95 19.72 323.89 19.46 323.89 L3.55 323.89 C3.16 323.89 2.82 324.02 2.54 324.23
+ C2.35 324.38 2.19 324.57 2.09 324.79 C1.98 325.01 1.92 325.25 1.92 325.52 L1.92 336.04 L21.09
+ 336.04 L21.09 325.52 ZM21.18 337.33 L1.77 337.33 L0 340.51 L0 342.55 L23.06 342.55 L23.06
+ 340.51 L21.18 337.33 Z" class="st9"/>
+ </g>
+ <path d="M21.09 325.52 C21.09 325.13 20.96 324.79 20.74 324.51 C20.59 324.32 20.4 324.16 20.19 324.06 C19.97
+ 323.95 19.72 323.89 19.46 323.89 L3.55 323.89 C3.16 323.89 2.82 324.02 2.54 324.23 C2.35 324.38
+ 2.19 324.57 2.09 324.79 C1.98 325.01 1.92 325.25 1.92 325.52 L1.92 336.04 L21.09 336.04 L21.09
+ 325.52 ZM21.18 337.33 L1.77 337.33 L0 340.51 L0 342.55 L23.06 342.55 L23.06 340.51 L21.18 337.33
+ Z" class="st28"/>
+ </g>
+ <g id="shape52-236" v:mID="52" v:groupContext="shape" v:layerMember="0" transform="translate(3.19243,-16.175)">
+ <title>Sheet.52</title>
+ <v:userDefs>
+ <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(#5b9bd5)"/>
+ </v:userDefs>
+ <path d="M16.62 342.55 L16.62 333.29 C16.62 333.23 16.61 333.18 16.58 333.13 C16.55 333.07 16.5 333.02 16.44
+ 332.98 C16.39 332.95 16.33 332.94 16.27 332.94 L0.35 332.94 C0.29 332.94 0.24 332.95 0.19 332.98
+ C0.13 333.01 0.08 333.07 0.04 333.12 C0.02 333.17 0 333.23 0 333.29 L0 342.55 L16.62 342.55
+ Z" class="st29"/>
+ </g>
+ <g id="shape53-238" v:mID="53" v:groupContext="shape" v:layerMember="0" transform="translate(1.97942,-10.81)">
+ <title>Sheet.53</title>
+ <v:userDefs>
+ <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(#5b9bd5)"/>
+ </v:userDefs>
+ <path d="M0.96 340.83 L0 342.55 L19.06 342.55 L18.1 340.83 L0.96 340.83 Z" class="st26"/>
+ </g>
+ </g>
+ <g id="group54-241" transform="translate(7.21495,-18.6146)" v:mID="54" v:groupContext="group" v:layerMember="0">
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ <v:ud v:nameU="ConnGap" v:prompt="" v:val="VT0(0.083333333333333):0"/>
+ </v:userDefs>
+ <title>User.12</title>
+ <g id="shape55-242" v:mID="55" v:groupContext="shape" v:layerMember="0"
+ transform="translate(13.3353,-1.13687E-013)">
+ <title>Sheet.55</title>
+ <g id="shadow55-243" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42 ZM20.96
+ 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52
+ 13.66 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55
+ L21.12 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36
+ 329.46 C22.2 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77 Z"
+ class="st24"/>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42"
+ class="st25"/>
+ <path d="M20.96 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52
+ 13.66 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55
+ L21.12 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36
+ 329.46 C22.2 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77"
+ class="st25"/>
+ <path d="M18.1 342.55 L18.1 338.37" class="st25"/>
+ <path d="M15.03 337.32 L15.03 333.71" class="st25"/>
+ <path d="M21.12 337.32 L21.12 333.71" class="st25"/>
+ <path d="M0 337.32 L13.47 337.32" class="st25"/>
+ </g>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42 ZM20.96
+ 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52 13.66
+ 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55 L21.12
+ 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36 329.46 C22.2
+ 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77 Z" class="st26"/>
+ <path d="M20.77 325.42 A2.63551 2.63601 -180 1 0 15.5 325.42 A2.63551 2.63601 -180 1 0 20.77 325.42"
+ class="st27"/>
+ <path d="M20.96 328.77 L15.25 328.77 C14.84 328.77 14.46 328.91 14.16 329.14 C13.95 329.31 13.78 329.52 13.66
+ 329.76 C13.54 330 13.47 330.27 13.47 330.55 L13.47 337.32 L15.03 337.32 L15.03 342.55 L21.12
+ 342.55 L21.12 337.32 L22.74 337.32 L22.74 330.55 C22.74 330.14 22.6 329.76 22.36 329.46 C22.2
+ 329.25 21.99 329.08 21.75 328.96 C21.51 328.84 21.24 328.77 20.96 328.77" class="st27"/>
+ <path d="M18.1 342.55 L18.1 338.37" class="st27"/>
+ <path d="M15.03 337.32 L15.03 333.71" class="st27"/>
+ <path d="M21.12 337.32 L21.12 333.71" class="st27"/>
+ <path d="M0 337.32 L13.47 337.32" class="st27"/>
+ </g>
+ <g id="shape56-258" v:mID="56" v:groupContext="shape" v:layerMember="0" transform="translate(0,-8.39743)">
+ <title>Sheet.56</title>
+ <g id="shadow56-259" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M21.09 325.52 C21.09 325.13 20.96 324.79 20.74 324.51 C20.59 324.32 20.4 324.16 20.19 324.06
+ C19.97 323.95 19.72 323.89 19.46 323.89 L3.55 323.89 C3.16 323.89 2.82 324.02 2.54 324.23
+ C2.35 324.38 2.19 324.57 2.09 324.79 C1.98 325.01 1.92 325.25 1.92 325.52 L1.92 336.04 L21.09
+ 336.04 L21.09 325.52 ZM21.18 337.33 L1.77 337.33 L0 340.51 L0 342.55 L23.06 342.55 L23.06
+ 340.51 L21.18 337.33 Z" class="st9"/>
+ </g>
+ <path d="M21.09 325.52 C21.09 325.13 20.96 324.79 20.74 324.51 C20.59 324.32 20.4 324.16 20.19 324.06 C19.97
+ 323.95 19.72 323.89 19.46 323.89 L3.55 323.89 C3.16 323.89 2.82 324.02 2.54 324.23 C2.35 324.38
+ 2.19 324.57 2.09 324.79 C1.98 325.01 1.92 325.25 1.92 325.52 L1.92 336.04 L21.09 336.04 L21.09
+ 325.52 ZM21.18 337.33 L1.77 337.33 L0 340.51 L0 342.55 L23.06 342.55 L23.06 340.51 L21.18 337.33
+ Z" class="st28"/>
+ </g>
+ <g id="shape57-262" v:mID="57" v:groupContext="shape" v:layerMember="0" transform="translate(3.19243,-16.175)">
+ <title>Sheet.57</title>
+ <v:userDefs>
+ <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(#5b9bd5)"/>
+ </v:userDefs>
+ <path d="M16.62 342.55 L16.62 333.29 C16.62 333.23 16.61 333.18 16.58 333.13 C16.55 333.07 16.5 333.02 16.44
+ 332.98 C16.39 332.95 16.33 332.94 16.27 332.94 L0.35 332.94 C0.29 332.94 0.24 332.95 0.19 332.98
+ C0.13 333.01 0.08 333.07 0.04 333.12 C0.02 333.17 0 333.23 0 333.29 L0 342.55 L16.62 342.55
+ Z" class="st29"/>
+ </g>
+ <g id="shape58-264" v:mID="58" v:groupContext="shape" v:layerMember="0" transform="translate(1.97942,-10.81)">
+ <title>Sheet.58</title>
+ <v:userDefs>
+ <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(#5b9bd5)"/>
+ </v:userDefs>
+ <path d="M0.96 340.83 L0 342.55 L19.06 342.55 L18.1 340.83 L0.96 340.83 Z" class="st26"/>
+ </g>
+ </g>
+ <g id="group59-267" transform="translate(171.161,-45.6707)" v:mID="59" v:groupContext="group" v:layerMember="0">
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ <v:ud v:nameU="ConnGap" v:prompt="" v:val="VT0(0.083333333333333):0"/>
+ </v:userDefs>
+ <title>Data Center</title>
+ <g id="shape60-268" v:mID="60" v:groupContext="shape" v:layerMember="0">
+ <title>Sheet.60</title>
+ <g id="shadow60-269" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <ellipse cx="37.8785" cy="331.299" rx="37.8785" ry="11.246" class="st9"/>
+ </g>
+ <ellipse cx="37.8785" cy="331.299" rx="37.8785" ry="11.246" class="st10"/>
+ </g>
+ <g id="shape61-272" v:mID="61" v:groupContext="shape" v:layerMember="0" transform="translate(6.86487,-7.30475)">
+ <title>Sheet.61</title>
+ <g id="shadow61-273" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M54.1 311.79 L43.28 311.79 L43.28 342.55 L62.03 342.55 L62.03 311.79 L54.1 311.79 ZM43.28 332.44
+ L43.28 311.79 L51.21 311.79 L51.21 301.69 L32.33 301.69 L32.33 311.79 L40.39 311.79 L40.39
+ 332.44 L43.28 332.44 ZM40.39 301.69 L40.39 293.03 L21.64 293.03 L21.64 301.69 L29.57 301.69
+ L29.57 311.79 L32.46 311.79 L32.46 301.69 L40.39 301.69 ZM32.46 311.79 L21.64 311.79 L21.64
+ 342.55 L40.39 342.55 L40.39 311.79 L32.46 311.79 ZM10.82 311.79 L0 311.79 L0 342.55 L18.75
+ 342.55 L18.75 311.79 L10.82 311.79 ZM21.64 311.79 L29.57 311.79 L29.57 301.69 L10.82 301.69
+ L10.82 311.79 L18.75 311.79 L18.75 332.44 L21.64 332.44 L21.64 311.79 Z" class="st9"/>
+ </g>
+ <path d="M54.1 311.79 L43.28 311.79 L43.28 342.55 L62.03 342.55 L62.03 311.79 L54.1 311.79 ZM43.28 332.44
+ L43.28 311.79 L51.21 311.79 L51.21 301.69 L32.33 301.69 L32.33 311.79 L40.39 311.79 L40.39 332.44
+ L43.28 332.44 ZM40.39 301.69 L40.39 293.03 L21.64 293.03 L21.64 301.69 L29.57 301.69 L29.57
+ 311.79 L32.46 311.79 L32.46 301.69 L40.39 301.69 ZM32.46 311.79 L21.64 311.79 L21.64 342.55
+ L40.39 342.55 L40.39 311.79 L32.46 311.79 ZM10.82 311.79 L0 311.79 L0 342.55 L18.75 342.55 L18.75
+ 311.79 L10.82 311.79 ZM21.64 311.79 L29.57 311.79 L29.57 301.69 L10.82 301.69 L10.82 311.79
+ L18.75 311.79 L18.75 332.44 L21.64 332.44 L21.64 311.79 Z" class="st10"/>
+ </g>
+ <g id="shape62-276" v:mID="62" v:groupContext="shape" v:layerMember="0" transform="translate(20.0835,-20.5174)">
+ <title>Sheet.62</title>
+ <g id="shadow62-277" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M45.36 341.36 A1.13296 1.18615 -180 1 0 43.09 341.36 A1.13296 1.18615 -180 1 0 45.36 341.36
+ ZM23.46 341.36 A1.13296 1.18615 -180 1 0 21.2 341.36 A1.13296 1.18615 -180 1 0 23.46 341.36
+ ZM2.27 341.36 A1.13296 1.18615 -180 1 0 0 341.36 A1.13296 1.18615 -180 1 0 2.27 341.36 Z"
+ class="st24"/>
+ </g>
+ <path d="M45.36 341.36 A1.13296 1.18615 -180 1 0 43.09 341.36 A1.13296 1.18615 -180 1 0 45.36 341.36 ZM23.46
+ 341.36 A1.13296 1.18615 -180 1 0 21.2 341.36 A1.13296 1.18615 -180 1 0 23.46 341.36 ZM2.27 341.36
+ A1.13296 1.18615 -180 1 0 0 341.36 A1.13296 1.18615 -180 1 0 2.27 341.36 Z" class="st30"/>
+ </g>
+ <g id="shape63-282" v:mID="63" v:groupContext="shape" v:layerMember="0" transform="translate(14.2717,-12.5134)">
+ <title>Sheet.63</title>
+ <v:userDefs>
+ <v:ud v:nameU="SurroundingRegionColor" v:prompt="" v:val="VT5(#5b9bd5)"/>
+ </v:userDefs>
+ <g id="shadow63-283" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M43.09 342.55 L51.17 342.55 L51.17 341.74 L43.09 341.74 L43.09 342.55 ZM43.09 340.12 L51.17
+ 340.12 L51.17 339.32 L43.09 339.32 L43.09 340.12 ZM43.09 337.69 L51.17 337.69 L51.17 336.89
+ L43.09 336.89 L43.09 337.69 ZM21.2 342.55 L29.27 342.55 L29.27 341.74 L21.2 341.74 L21.2
+ 342.55 ZM21.2 340.12 L29.27 340.12 L29.27 339.32 L21.2 339.32 L21.2 340.12 ZM21.2 337.69
+ L29.27 337.69 L29.27 336.89 L21.2 336.89 L21.2 337.69 ZM-0 342.55 L8.08 342.55 L8.08 341.74
+ L-0 341.74 L-0 342.55 ZM-0 340.12 L8.08 340.12 L8.08 339.32 L-0 339.32 L-0 340.12 ZM-0 337.69
+ L8.08 337.69 L8.08 336.89 L-0 336.89 L-0 337.69 Z" class="st24"/>
+ </g>
+ <path d="M43.09 342.55 L51.17 342.55 L51.17 341.74 L43.09 341.74 L43.09 342.55 ZM43.09 340.12 L51.17 340.12
+ L51.17 339.32 L43.09 339.32 L43.09 340.12 ZM43.09 337.69 L51.17 337.69 L51.17 336.89 L43.09
+ 336.89 L43.09 337.69 ZM21.2 342.55 L29.27 342.55 L29.27 341.74 L21.2 341.74 L21.2 342.55 ZM21.2
+ 340.12 L29.27 340.12 L29.27 339.32 L21.2 339.32 L21.2 340.12 ZM21.2 337.69 L29.27 337.69 L29.27
+ 336.89 L21.2 336.89 L21.2 337.69 ZM-0 342.55 L8.08 342.55 L8.08 341.74 L-0 341.74 L-0 342.55
+ ZM-0 340.12 L8.08 340.12 L8.08 339.32 L-0 339.32 L-0 340.12 ZM-0 337.69 L8.08 337.69 L8.08 336.89
+ L-0 336.89 L-0 337.69 Z" class="st26"/>
+ </g>
+ </g>
+ <g id="group64-288" transform="translate(59.5234,-47.1858)" v:mID="64" v:groupContext="group">
+ <v:custProps>
+ <v:cp v:nameU="AssetNumber" v:lbl="Asset Number" v:prompt="" v:type="0" v:format="" v:sortKey="Asset"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="SerialNumber" v:lbl="Serial Number" v:prompt="" v:type="0" v:format="" v:sortKey="Asset"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="Location" v:lbl="Location" v:prompt="" v:type="0" v:format="" v:sortKey="Asset"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="Building" v:lbl="Building" v:prompt="" v:type="0" v:format="" v:sortKey="Asset"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="Room" v:lbl="Room" v:prompt="" v:type="0" v:format="" v:sortKey="Asset" v:invis="false"
+ v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="Manufacturer" v:lbl="Manufacturer" v:prompt="" v:type="0" v:format="" v:sortKey="Equipment"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="ProductNumber" v:lbl="Product Number" v:prompt="" v:type="0" v:format=""
+ v:sortKey="Equipment" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="PartNumber" v:lbl="Part Number" v:prompt="" v:type="0" v:format="" v:sortKey="Equipment"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="ProductDescription" v:lbl="Product Description" v:prompt="" v:type="0" v:format=""
+ v:sortKey="Equipment" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="NetworkName" v:lbl="Network Name" v:prompt="" v:type="0" v:format="" v:sortKey="Network"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="IPAddress" v:lbl="IP Address" v:prompt="" v:type="0" v:format="" v:sortKey="Network"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="SubnetMask" v:lbl="Subnet Mask" v:prompt="" v:type="0" v:format="" v:sortKey="Network"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="AdminInterface" v:lbl="Administrative Interface" v:prompt="" v:type="0" v:format=""
+ v:sortKey="Network" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="NumberofPorts" v:lbl="Number of Ports" v:prompt="" v:type="0" v:format=""
+ v:sortKey="Network" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="CommunityString" v:lbl="Community String" v:prompt="" v:type="0" v:format=""
+ v:sortKey="Network" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="NetworkDescription" v:lbl="Network Description" v:prompt="" v:type="0" v:format=""
+ v:sortKey="Network" v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="MACAddress" v:lbl="MAC Address" v:prompt="" v:type="0" v:format="" v:sortKey="Network"
+ v:invis="false" v:ask="false" v:langID="1033" v:cal="0"/>
+ <v:cp v:nameU="ShapeClass" v:lbl="ShapeClass" v:prompt="" v:type="0" v:format="" v:sortKey=""
+ v:invis="true" v:ask="false" v:langID="1033" v:cal="0" v:val="VT4(Equipment)"/>
+ <v:cp v:nameU="ShapeType" v:lbl="ShapeType" v:prompt="" v:type="0" v:format="" v:sortKey="" v:invis="true"
+ v:ask="false" v:langID="1033" v:cal="0" v:val="VT4(Device)"/>
+ <v:cp v:nameU="SubShapeType" v:lbl="SubShapeType" v:prompt="" v:type="0" v:format="" v:sortKey=""
+ v:invis="true" v:ask="false" v:langID="1033" v:cal="0" v:val="VT4(Load balancer)"/>
+ </v:custProps>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ <v:ud v:nameU="ShapeClass" v:prompt="" v:val="VT0(5):26"/>
+ <v:ud v:nameU="SolSH" v:prompt="" v:val="VT14({BF0433D9-CD73-4EB5-8390-8653BE590246}):41"/>
+ <v:ud v:nameU="visLegendShape" v:prompt="" v:val="VT0(2):26"/>
+ </v:userDefs>
+ <title>Load balancer</title>
+ <g id="shape65-289" v:mID="65" v:groupContext="shape" transform="translate(0,-1.653)">
+ <title>Sheet.65</title>
+ <g id="shadow65-290" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M12.18 329.62 L4.06 329.62 L0 332.02 L0 342.55 L16.23 342.55 L16.23 332.02 L12.18 329.62 Z"
+ class="st24"/>
+ <path d="M0 332.02 L16.23 332.02" class="st25"/>
+ <path d="M12.18 329.62 L4.06 329.62 L0 332.02 L0 342.55 L16.23 342.55 L16.23 332.02 L12.18 329.62"
+ class="st25"/>
+ </g>
+ <path d="M12.18 329.62 L4.06 329.62 L0 332.02 L0 342.55 L16.23 342.55 L16.23 332.02 L12.18 329.62 Z"
+ class="st30"/>
+ <path d="M0 332.02 L16.23 332.02" class="st31"/>
+ <path d="M12.18 329.62 L4.06 329.62 L0 332.02 L0 342.55 L16.23 342.55 L16.23 332.02 L12.18 329.62"
+ class="st31"/>
+ </g>
+ <g id="shape66-297" v:mID="66" v:groupContext="shape" transform="translate(1.81062,-2.91583)">
+ <title>Sheet.66</title>
+ <g id="shadow66-298" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M10.22 341.92 L9.29 342.12 L9.95 342.55 L11.2 342.23 L10.99 340.96 L10.33 340.52 L10.53 341.44
+ L8.34 340.01 L8.03 340.49 L10.22 341.92 ZM11.46 338.22 L8.84 338.22 L8.84 338.78 L11.45
+ 338.78 L10.78 339.45 L11.57 339.45 L12.45 338.5 L11.57 337.55 L10.78 337.55 L11.46 338.22
+ ZM10.48 335.2 L8.29 336.64 L8.6 337.12 L10.79 335.68 L10.59 336.61 L11.25 336.17 L11.46
+ 334.9 L10.21 334.58 L9.55 335.01 L10.48 335.2 ZM6.25 336.37 C5.11 336.37 4.19 337.29 4.19
+ 338.43 C4.19 339.56 5.11 340.48 6.25 340.48 C7.38 340.48 8.31 339.56 8.31 338.43 C8.31 337.29
+ 7.38 336.37 6.25 336.37 ZM6.25 337.02 C7.02 337.02 7.66 337.65 7.66 338.43 C7.66 339.2 7.02
+ 339.83 6.25 339.83 C5.47 339.83 4.84 339.2 4.84 338.43 C4.84 337.65 5.47 337.02 6.25 337.02
+ ZM2.62 338.14 L0 338.14 L0 338.71 L2.62 338.71 L1.94 339.38 L2.74 339.38 L3.61 338.43 L2.73
+ 337.47 L1.95 337.47 L2.62 338.14 Z" class="st9"/>
+ </g>
+ <path d="M10.22 341.92 L9.29 342.12 L9.95 342.55 L11.2 342.23 L10.99 340.96 L10.33 340.52 L10.53 341.44 L8.34
+ 340.01 L8.03 340.49 L10.22 341.92 ZM11.46 338.22 L8.84 338.22 L8.84 338.78 L11.45 338.78 L10.78
+ 339.45 L11.57 339.45 L12.45 338.5 L11.57 337.55 L10.78 337.55 L11.46 338.22 ZM10.48 335.2 L8.29
+ 336.64 L8.6 337.12 L10.79 335.68 L10.59 336.61 L11.25 336.17 L11.46 334.9 L10.21 334.58 L9.55
+ 335.01 L10.48 335.2 ZM6.25 336.37 C5.11 336.37 4.19 337.29 4.19 338.43 C4.19 339.56 5.11 340.48
+ 6.25 340.48 C7.38 340.48 8.31 339.56 8.31 338.43 C8.31 337.29 7.38 336.37 6.25 336.37 ZM6.25
+ 337.02 C7.02 337.02 7.66 337.65 7.66 338.43 C7.66 339.2 7.02 339.83 6.25 339.83 C5.47 339.83
+ 4.84 339.2 4.84 338.43 C4.84 337.65 5.47 337.02 6.25 337.02 ZM2.62 338.14 L0 338.14 L0 338.71
+ L2.62 338.71 L1.94 339.38 L2.74 339.38 L3.61 338.43 L2.73 337.47 L1.95 337.47 L2.62 338.14 Z"
+ class="st32"/>
+ </g>
+ </g>
+ <g id="group67-301" transform="translate(104.617,-86.5795)" v:mID="67" v:groupContext="group">
+ <v:userDefs>
+ <v:ud v:nameU="SkinColor" v:prompt="" v:val="VT5(#da8c36)"/>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <title>Directory server</title>
+ <g id="shape68-302" v:mID="68" v:groupContext="shape" transform="translate(0,-0.451005)">
+ <title>Sheet.68</title>
+ <g id="shadow68-303" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0.47 329.86 L0.47 331.94 L1.46 332.57 L3.33 331.52 L15.43 338.57 L15.43 340.61 L16.42 341.24
+ L18.24 340.22 L22.24 342.55 L22.24 339.27 L36.07 331.28 L36.07 321.27 L19.64 311.85 L3.16
+ 321.13 L3.16 321.5 L0 319.68 L0 329.58 L0.47 329.86 Z" class="st33"/>
+ </g>
+ <path d="M0.47 329.86 L0.47 331.94 L1.46 332.57 L3.33 331.52 L15.43 338.57 L15.43 340.61 L16.42 341.24 L18.24
+ 340.22 L22.24 342.55 L22.24 339.27 L36.07 331.28 L36.07 321.27 L19.64 311.85 L3.16 321.13 L3.16
+ 321.5 L0 319.68 L0 329.58 L0.47 329.86 Z" class="st34"/>
+ </g>
+ <g id="shape69-306" v:mID="69" v:groupContext="shape" transform="translate(3.1636,-11.8063)">
+ <title>Sheet.69</title>
+ <path d="M16.48 323.24 L32.91 332.66 L16.31 342.55 L0 333.26 L0 332.52 L16.48 323.24 Z" class="st35"/>
+ </g>
+ <g id="shape70-310" v:mID="70" v:groupContext="shape" transform="translate(19.06,-3.68954)">
+ <title>Sheet.70</title>
+ <path d="M17.01 324.55 L0 334.19 L3.18 342.55 L17.01 334.56 L17.01 324.55 Z" class="st36"/>
+ </g>
+ <g id="shape71-314" v:mID="71" v:groupContext="shape" transform="translate(0,-0.415652)">
+ <title>Sheet.71</title>
+ <path d="M22.24 342.55 L0 329.58 L0 319.68 L22.24 332.43 L22.24 342.55 Z" class="st37"/>
+ </g>
+ <g id="shape72-322" v:mID="72" v:groupContext="shape" transform="translate(0.82443,-19.8334)">
+ <title>Sheet.72</title>
+ <path d="M1.13 341.58 a0.653986 0.653986 -180 0 0 -0.73971 -0.492434 a0.656072 0.656072 -180 0 0 -0.253101
+ 0.865731 a0.653066 0.653066 -180 0 0 0.740769 0.491375 a0.655459 0.655459 -180 0 0 0.252042
+ -0.864672 Z" class="st38"/>
+ </g>
+ <g id="shape73-326" v:mID="73" v:groupContext="shape" transform="translate(3.62283,-15.1638)">
+ <title>Sheet.73</title>
+ <path d="M3.22 339.78 A1.86495 1.86495 -180 0 0 1.11 338.38 A1.8709 1.8709 -180 0 0 0.38 340.85 A1.86532
+ 1.86532 -180 0 0 2.5 342.25 A1.87264 1.87264 -180 0 0 3.22 339.78 Z" class="st38"/>
+ </g>
+ <g id="shape74-329" v:mID="74" v:groupContext="shape" transform="translate(3.62283,-10.4867)">
+ <title>Sheet.74</title>
+ <path d="M3.22 339.78 A1.86495 1.86495 -180 0 0 1.11 338.38 A1.8709 1.8709 -180 0 0 0.38 340.85 A1.86532
+ 1.86532 -180 0 0 2.5 342.25 A1.87264 1.87264 -180 0 0 3.22 339.78 Z" class="st38"/>
+ </g>
+ <g id="shape75-332" v:mID="75" v:groupContext="shape" transform="translate(4.52404,-16.3668)">
+ <title>Sheet.75</title>
+ <path d="M1.61 341.16 a0.931952 0.931952 -180 0 0 -1.05741 -0.702645 a0.935408 0.935408 -180 0 0 -0.361118
+ 1.23585 a0.932139 0.932139 -180 0 0 1.05794 0.702645 a0.935822 0.935822 -180 0 0 0.360589 -1.23585
+ Z" class="st39"/>
+ </g>
+ <g id="shape76-336" v:mID="76" v:groupContext="shape" transform="translate(4.52404,-11.6897)">
+ <title>Sheet.76</title>
+ <path d="M1.61 341.16 a0.931952 0.931952 -180 0 0 -1.05741 -0.702645 a0.935875 0.935875 -180 0 0 -0.361118
+ 1.23585 a0.932139 0.932139 -180 0 0 1.05794 0.702645 a0.935822 0.935822 -180 0 0 0.360589 -1.23585
+ Z" class="st39"/>
+ </g>
+ <g id="shape77-339" v:mID="77" v:groupContext="shape" transform="translate(7.78787,-8.83469)">
+ <title>Sheet.77</title>
+ <path d="M0 341.57 L0.05 333.6 L1.83 334.57 L1.78 342.55 L0 341.57 Z" class="st40"/>
+ </g>
+ <g id="shape78-343" v:mID="78" v:groupContext="shape" transform="translate(10.204,-7.4008)">
+ <title>Sheet.78</title>
+ <path d="M0 341.57 L0.05 333.6 L1.83 334.57 L1.78 342.55 L0 341.57 Z" class="st40"/>
+ </g>
+ <g id="shape79-346" v:mID="79" v:groupContext="shape" transform="translate(12.6196,-5.96639)">
+ <title>Sheet.79</title>
+ <path d="M0 341.57 L0.05 333.6 L1.83 334.57 L1.78 342.55 L0 341.57 Z" class="st40"/>
+ </g>
+ <g id="shape80-349" v:mID="80" v:groupContext="shape" transform="translate(15.0357,-4.53251)">
+ <title>Sheet.80</title>
+ <path d="M0 341.57 L0.05 333.6 L1.83 334.57 L1.78 342.55 L0 341.57 Z" class="st40"/>
+ </g>
+ <g id="shape81-352" v:mID="81" v:groupContext="shape" transform="translate(8.24006,-10.0631)">
+ <title>Sheet.81</title>
+ <path d="M0.85 342.24 a0.388199 0.388199 0 0 1 -0.425188 0.308698 a0.638045 0.638045 0 0 1 -0.424658 -0.573447
+ L0 336.5 a0.387575 0.387575 0 0 1 0.424658 -0.308698 a0.637725 0.637725 0 0 1 0.425188 0.573447
+ L0.85 342.24 Z" class="st41"/>
+ </g>
+ <g id="shape82-356" v:mID="82" v:groupContext="shape" transform="translate(10.6556,-8.62924)">
+ <title>Sheet.82</title>
+ <path d="M0.85 342.24 a0.388199 0.388199 0 0 1 -0.425188 0.308698 a0.638045 0.638045 0 0 1 -0.424658 -0.573447
+ L0 336.5 a0.387575 0.387575 0 0 1 0.424658 -0.308698 a0.637725 0.637725 0 0 1 0.425188 0.573447
+ L0.85 342.24 Z" class="st41"/>
+ </g>
+ <g id="shape83-359" v:mID="83" v:groupContext="shape" transform="translate(13.0717,-7.19483)">
+ <title>Sheet.83</title>
+ <path d="M0.85 342.24 a0.388199 0.388199 0 0 1 -0.425188 0.308698 a0.638045 0.638045 0 0 1 -0.424658 -0.573447
+ L0 336.5 a0.387575 0.387575 0 0 1 0.424658 -0.308698 a0.637725 0.637725 0 0 1 0.425188 0.573447
+ L0.85 342.24 Z" class="st41"/>
+ </g>
+ <g id="shape84-362" v:mID="84" v:groupContext="shape" transform="translate(15.4873,-5.76095)">
+ <title>Sheet.84</title>
+ <path d="M0.85 342.24 a0.388502 0.388502 0 0 1 -0.425717 0.308698 a0.638367 0.638367 0 0 1 -0.424129 -0.573447
+ L0 336.5 a0.387272 0.387272 0 0 1 0.424129 -0.308698 a0.638235 0.638235 0 0 1 0.425717 0.573447
+ L0.85 342.24 Z" class="st41"/>
+ </g>
+ <g id="shape85-365" v:mID="85" v:groupContext="shape" transform="translate(7.78787,-9.81214)">
+ <title>Sheet.85</title>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55 L0 342.55 Z" class="st42"/>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55" class="st43"/>
+ </g>
+ <g id="shape86-368" v:mID="86" v:groupContext="shape" transform="translate(10.204,-8.37826)">
+ <title>Sheet.86</title>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55 L0 342.55 Z" class="st42"/>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55" class="st43"/>
+ </g>
+ <g id="shape87-371" v:mID="87" v:groupContext="shape" transform="translate(12.6196,-6.94385)">
+ <title>Sheet.87</title>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55 L0 342.55 Z" class="st42"/>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55" class="st43"/>
+ </g>
+ <g id="shape88-374" v:mID="88" v:groupContext="shape" transform="translate(15.0357,-5.50996)">
+ <title>Sheet.88</title>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55 L0 342.55 Z" class="st42"/>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55" class="st43"/>
+ </g>
+ <g id="shape89-377" v:mID="89" v:groupContext="shape" transform="translate(7.78787,-4.53251)">
+ <title>Sheet.89</title>
+ <path d="M9.08 334.57 L9.03 342.55 L7.25 341.57 L9.08 334.57 ZM6.66 333.14 L6.61 341.11 L4.83 340.13 L6.66
+ 333.14 ZM4.25 331.7 L4.2 339.68 L2.42 338.7 L4.25 331.7 ZM1.83 330.27 L1.78 338.24 L0 337.27
+ L1.83 330.27 Z" class="st42"/>
+ <path d="M9.08 334.57 L9.03 342.55 L7.25 341.57M6.66 333.14 L6.61 341.11 L4.83 340.13M4.25 331.7 L4.2 339.68
+ L2.42 338.7M1.83 330.27 L1.78 338.24 L0 337.27" class="st44"/>
+ </g>
+ <g id="shape90-380" v:mID="90" v:groupContext="shape" transform="translate(2.22125,-11.8454)">
+ <title>Sheet.90</title>
+ <path d="M0 341.85 L0.63 341.42 L1.42 341.78 L0.03 342.55 L0 341.85 Z" class="st45"/>
+ </g>
+ <g id="shape91-384" v:mID="91" v:groupContext="shape" transform="translate(17.1796,-3.17487)">
+ <title>Sheet.91</title>
+ <path d="M0 341.85 L0.63 341.42 L1.42 341.78 L0.03 342.55 L0 341.85 Z" class="st45"/>
+ </g>
+ <g id="shape92-387" v:mID="92" v:groupContext="shape" transform="translate(1.46036,-10.3893)">
+ <title>Sheet.92</title>
+ <path d="M2.12 341.35 L0 342.55 L0 333.54 L2.12 332.29 L2.12 333.41 L0.79 334.15 L0.79 341.09 L2.18 340.33
+ L2.12 341.35 Z" class="st36"/>
+ </g>
+ <g id="shape93-390" v:mID="93" v:groupContext="shape" transform="translate(16.4187,-1.71875)">
+ <title>Sheet.93</title>
+ <path d="M2.12 341.35 L0 342.55 L0 333.54 L2.12 332.29 L2.12 333.41 L0.79 334.15 L0.79 341.09 L2.18 340.33
+ L2.12 341.35 Z" class="st36"/>
+ </g>
+ <g id="shape94-393" v:mID="94" v:groupContext="shape" transform="translate(0.467548,-10.3893)">
+ <title>Sheet.94</title>
+ <path d="M0.99 333.54 L3.11 332.29 L2.12 331.66 L0 332.91 L0 341.92 L0.99 342.55 L0.99 333.54 Z"
+ class="st46"/>
+ </g>
+ <g id="shape95-397" v:mID="95" v:groupContext="shape" transform="translate(15.4259,-1.71875)">
+ <title>Sheet.95</title>
+ <path d="M0.99 333.54 L3.11 332.29 L2.12 331.66 L0 332.91 L0 341.92 L0.99 342.55 L0.99 333.54 Z"
+ class="st46"/>
+ </g>
+ <g id="shape96-400" v:mID="96" v:groupContext="shape" transform="translate(0.467548,-1.71928)">
+ <title>Sheet.96</title>
+ <path d="M17.34 339.96 L16.75 340.37 L16.75 334.15 L18.07 333.41 L18.07 332.29 L17.08 331.66 L14.96 332.91
+ L14.96 341.92 L15.95 342.55 L18.07 341.35 L18.14 340.33 L17.34 339.96 ZM2.38 331.29 L1.79 331.7
+ L1.79 325.48 L3.11 324.74 L3.11 323.62 L2.12 322.99 L0 324.24 L0 333.25 L0.99 333.87 L3.11 332.68
+ L3.18 331.66 L2.38 331.29 Z" class="st47"/>
+ </g>
+ <g id="shape97-402" v:mID="97" v:groupContext="shape" transform="translate(19.9526,-8.71396)">
+ <title>Sheet.97</title>
+ <path d="M1.13 341.58 a0.653986 0.653986 -180 0 0 -0.73971 -0.492434 a0.656072 0.656072 -180 0 0 -0.253101
+ 0.865731 a0.653066 0.653066 -180 0 0 0.740769 0.491375 a0.655459 0.655459 -180 0 0 0.252042
+ -0.864672 Z" class="st38"/>
+ </g>
+ <g id="shape98-405" v:mID="98" v:groupContext="shape" transform="translate(19.9526,-2.35997)">
+ <title>Sheet.98</title>
+ <path d="M1.13 341.58 a0.653986 0.653986 -180 0 0 -0.73971 -0.492434 a0.656072 0.656072 -180 0 0 -0.253101
+ 0.865731 a0.653066 0.653066 -180 0 0 0.740769 0.491375 a0.655459 0.655459 -180 0 0 0.252042
+ -0.864672 Z" class="st38"/>
+ </g>
+ <g id="shape99-408" v:mID="99" v:groupContext="shape" transform="translate(0,-0.415652)">
+ <title>Sheet.99</title>
+ <path d="M36.07 331.28 L36.07 321.27 L19.64 311.85 L3.16 321.13 L3.16 321.52 L0 319.68 L0 329.58 L0.47 329.86
+ L0.47 331.94 L1.46 332.57 L3.33 331.52 L15.43 338.57 L15.43 340.61 L16.42 341.24 L18.24 340.22
+ L22.24 342.55 L22.24 339.27 L36.07 331.28 Z" class="st47"/>
+ </g>
+ <g id="shape100-410" v:mID="100" v:groupContext="shape" transform="translate(27.8077,-2.86477)">
+ <title>Sheet.100</title>
+ <path d="M0.29 342.55 L6.62 338.89 A1.82805 1.82805 0 0 1 6.62 336.9 L0.29 340.55 A1.82805 1.82805 -180 0
+ 0 0.29 342.55 Z" class="st48"/>
+ </g>
+ <g id="shape101-412" v:mID="101" v:groupContext="shape" transform="translate(23.5035,-4.85627)">
+ <title>Sheet.101</title>
+ <path d="M4.6 342.55 L10.92 338.89 L6.32 336.24 L0 339.89 L4.6 342.55 Z" class="st49"/>
+ </g>
+ <g id="shape102-416" v:mID="102" v:groupContext="shape" transform="translate(23.3588,-2.86477)">
+ <title>Sheet.102</title>
+ <path d="M0.14 339.89 L4.74 342.55 A1.82805 1.82805 0 0 1 4.74 340.55 L0.14 337.9 A3.49826 3.49826 -180 0
+ 0 0.14 339.89 Z" class="st50"/>
+ </g>
+ <g id="shape103-420" v:mID="103" v:groupContext="shape" transform="translate(25.8933,-5.98478)">
+ <title>Sheet.103</title>
+ <path d="M2.87 342.55 L0 340.89" class="st51"/>
+ <path d="M0.94 340.34 L3.82 342" class="st51"/>
+ <path d="M1.88 339.8 L4.76 341.46" class="st51"/>
+ <path d="M2.82 339.26 L5.7 340.92" class="st51"/>
+ <path d="M3.76 338.71 L6.64 340.37" class="st51"/>
+ </g>
+ <g id="shape104-427" v:mID="104" v:groupContext="shape" transform="translate(23.5035,-7.51159)">
+ <title>Sheet.104</title>
+ <path d="M5.13 341.17 L11.45 337.52 A11.9345 11.9345 0 0 1 6.32 338.89 L0 342.55 A11.9345 11.9345 -180 0
+ 0 5.13 341.17 Z" class="st52"/>
+ </g>
+ <g id="shape105-435" v:mID="105" v:groupContext="shape" transform="translate(30.2106,-4.74563)">
+ <title>Sheet.105</title>
+ <path d="M0.98 341.98 L0 342.55" class="st51"/>
+ <path d="M1.26 341.48 L2.24 340.92" class="st51"/>
+ <path d="M2.53 340.42 L3.51 339.86" class="st51"/>
+ </g>
+ <g id="shape106-440" v:mID="106" v:groupContext="shape" transform="translate(23.3588,-2.86477)">
+ <title>Sheet.106</title>
+ <path d="M0.14 339.89 L4.74 342.55 L11.07 338.89 A1.82805 1.82805 0 0 1 11.07 336.9 L7.85 335.04 L11.6 332.87
+ A11.9345 11.9345 0 0 1 6.47 334.25 L0.14 337.9 A3.49826 3.49826 -180 0 0 0.14 339.89"
+ class="st53"/>
+ </g>
+ </g>
+ <g id="group107-443" transform="translate(104.617,-33.8201)" v:mID="107" v:groupContext="group">
+ <v:userDefs>
+ <v:ud v:nameU="SkinColor" v:prompt="" v:val="VT5(#da8c36)"/>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <title>Directory server.104</title>
+ <g id="shape108-444" v:mID="108" v:groupContext="shape" transform="translate(0,-0.451005)">
+ <title>Sheet.108</title>
+ <g id="shadow108-445" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M0.47 329.86 L0.47 331.94 L1.46 332.57 L3.33 331.52 L15.43 338.57 L15.43 340.61 L16.42 341.24
+ L18.24 340.22 L22.24 342.55 L22.24 339.27 L36.07 331.28 L36.07 321.27 L19.64 311.85 L3.16
+ 321.13 L3.16 321.5 L0 319.68 L0 329.58 L0.47 329.86 Z" class="st33"/>
+ </g>
+ <path d="M0.47 329.86 L0.47 331.94 L1.46 332.57 L3.33 331.52 L15.43 338.57 L15.43 340.61 L16.42 341.24 L18.24
+ 340.22 L22.24 342.55 L22.24 339.27 L36.07 331.28 L36.07 321.27 L19.64 311.85 L3.16 321.13 L3.16
+ 321.5 L0 319.68 L0 329.58 L0.47 329.86 Z" class="st34"/>
+ </g>
+ <g id="shape109-448" v:mID="109" v:groupContext="shape" transform="translate(3.1636,-11.8063)">
+ <title>Sheet.109</title>
+ <path d="M16.48 323.24 L32.91 332.66 L16.31 342.55 L0 333.26 L0 332.52 L16.48 323.24 Z" class="st35"/>
+ </g>
+ <g id="shape110-451" v:mID="110" v:groupContext="shape" transform="translate(19.06,-3.68954)">
+ <title>Sheet.110</title>
+ <path d="M17.01 324.55 L0 334.19 L3.18 342.55 L17.01 334.56 L17.01 324.55 Z" class="st36"/>
+ </g>
+ <g id="shape111-454" v:mID="111" v:groupContext="shape" transform="translate(0,-0.415652)">
+ <title>Sheet.111</title>
+ <path d="M22.24 342.55 L0 329.58 L0 319.68 L22.24 332.43 L22.24 342.55 Z" class="st37"/>
+ </g>
+ <g id="shape112-457" v:mID="112" v:groupContext="shape" transform="translate(0.82443,-19.8334)">
+ <title>Sheet.112</title>
+ <path d="M1.13 341.58 a0.653986 0.653986 -180 0 0 -0.73971 -0.492434 a0.656072 0.656072 -180 0 0 -0.253101
+ 0.865731 a0.653066 0.653066 -180 0 0 0.740769 0.491375 a0.655459 0.655459 -180 0 0 0.252042
+ -0.864672 Z" class="st38"/>
+ </g>
+ <g id="shape113-460" v:mID="113" v:groupContext="shape" transform="translate(3.62283,-15.1638)">
+ <title>Sheet.113</title>
+ <path d="M3.22 339.78 A1.86495 1.86495 -180 0 0 1.11 338.38 A1.8709 1.8709 -180 0 0 0.38 340.85 A1.86532
+ 1.86532 -180 0 0 2.5 342.25 A1.87264 1.87264 -180 0 0 3.22 339.78 Z" class="st38"/>
+ </g>
+ <g id="shape114-463" v:mID="114" v:groupContext="shape" transform="translate(3.62283,-10.4867)">
+ <title>Sheet.114</title>
+ <path d="M3.22 339.78 A1.86495 1.86495 -180 0 0 1.11 338.38 A1.8709 1.8709 -180 0 0 0.38 340.85 A1.86532
+ 1.86532 -180 0 0 2.5 342.25 A1.87264 1.87264 -180 0 0 3.22 339.78 Z" class="st38"/>
+ </g>
+ <g id="shape115-466" v:mID="115" v:groupContext="shape" transform="translate(4.52404,-16.3668)">
+ <title>Sheet.115</title>
+ <path d="M1.61 341.16 a0.931952 0.931952 -180 0 0 -1.05741 -0.702645 a0.935408 0.935408 -180 0 0 -0.361118
+ 1.23585 a0.932139 0.932139 -180 0 0 1.05794 0.702645 a0.935822 0.935822 -180 0 0 0.360589 -1.23585
+ Z" class="st39"/>
+ </g>
+ <g id="shape116-469" v:mID="116" v:groupContext="shape" transform="translate(4.52404,-11.6897)">
+ <title>Sheet.116</title>
+ <path d="M1.61 341.16 a0.931952 0.931952 -180 0 0 -1.05741 -0.702645 a0.935875 0.935875 -180 0 0 -0.361118
+ 1.23585 a0.932139 0.932139 -180 0 0 1.05794 0.702645 a0.935822 0.935822 -180 0 0 0.360589 -1.23585
+ Z" class="st39"/>
+ </g>
+ <g id="shape117-472" v:mID="117" v:groupContext="shape" transform="translate(7.78787,-8.83469)">
+ <title>Sheet.117</title>
+ <path d="M0 341.57 L0.05 333.6 L1.83 334.57 L1.78 342.55 L0 341.57 Z" class="st40"/>
+ </g>
+ <g id="shape118-475" v:mID="118" v:groupContext="shape" transform="translate(10.204,-7.4008)">
+ <title>Sheet.118</title>
+ <path d="M0 341.57 L0.05 333.6 L1.83 334.57 L1.78 342.55 L0 341.57 Z" class="st40"/>
+ </g>
+ <g id="shape119-478" v:mID="119" v:groupContext="shape" transform="translate(12.6196,-5.96639)">
+ <title>Sheet.119</title>
+ <path d="M0 341.57 L0.05 333.6 L1.83 334.57 L1.78 342.55 L0 341.57 Z" class="st40"/>
+ </g>
+ <g id="shape120-481" v:mID="120" v:groupContext="shape" transform="translate(15.0357,-4.53251)">
+ <title>Sheet.120</title>
+ <path d="M0 341.57 L0.05 333.6 L1.83 334.57 L1.78 342.55 L0 341.57 Z" class="st40"/>
+ </g>
+ <g id="shape121-484" v:mID="121" v:groupContext="shape" transform="translate(8.24006,-10.0631)">
+ <title>Sheet.121</title>
+ <path d="M0.85 342.24 a0.388199 0.388199 0 0 1 -0.425188 0.308698 a0.638045 0.638045 0 0 1 -0.424658 -0.573447
+ L0 336.5 a0.387575 0.387575 0 0 1 0.424658 -0.308698 a0.637725 0.637725 0 0 1 0.425188 0.573447
+ L0.85 342.24 Z" class="st41"/>
+ </g>
+ <g id="shape122-487" v:mID="122" v:groupContext="shape" transform="translate(10.6556,-8.62924)">
+ <title>Sheet.122</title>
+ <path d="M0.85 342.24 a0.388199 0.388199 0 0 1 -0.425188 0.308698 a0.638045 0.638045 0 0 1 -0.424658 -0.573447
+ L0 336.5 a0.387575 0.387575 0 0 1 0.424658 -0.308698 a0.637725 0.637725 0 0 1 0.425188 0.573447
+ L0.85 342.24 Z" class="st41"/>
+ </g>
+ <g id="shape123-490" v:mID="123" v:groupContext="shape" transform="translate(13.0717,-7.19483)">
+ <title>Sheet.123</title>
+ <path d="M0.85 342.24 a0.388199 0.388199 0 0 1 -0.425188 0.308698 a0.638045 0.638045 0 0 1 -0.424658 -0.573447
+ L0 336.5 a0.387575 0.387575 0 0 1 0.424658 -0.308698 a0.637725 0.637725 0 0 1 0.425188 0.573447
+ L0.85 342.24 Z" class="st41"/>
+ </g>
+ <g id="shape124-493" v:mID="124" v:groupContext="shape" transform="translate(15.4873,-5.76095)">
+ <title>Sheet.124</title>
+ <path d="M0.85 342.24 a0.388502 0.388502 0 0 1 -0.425717 0.308698 a0.638367 0.638367 0 0 1 -0.424129 -0.573447
+ L0 336.5 a0.387272 0.387272 0 0 1 0.424129 -0.308698 a0.638235 0.638235 0 0 1 0.425717 0.573447
+ L0.85 342.24 Z" class="st41"/>
+ </g>
+ <g id="shape125-496" v:mID="125" v:groupContext="shape" transform="translate(7.78787,-9.81214)">
+ <title>Sheet.125</title>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55 L0 342.55 Z" class="st42"/>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55" class="st43"/>
+ </g>
+ <g id="shape126-499" v:mID="126" v:groupContext="shape" transform="translate(10.204,-8.37826)">
+ <title>Sheet.126</title>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55 L0 342.55 Z" class="st42"/>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55" class="st43"/>
+ </g>
+ <g id="shape127-502" v:mID="127" v:groupContext="shape" transform="translate(12.6196,-6.94385)">
+ <title>Sheet.127</title>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55 L0 342.55 Z" class="st42"/>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55" class="st43"/>
+ </g>
+ <g id="shape128-505" v:mID="128" v:groupContext="shape" transform="translate(15.0357,-5.50996)">
+ <title>Sheet.128</title>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55 L0 342.55 Z" class="st42"/>
+ <path d="M0 342.55 L0.05 334.57 L1.83 335.55" class="st43"/>
+ </g>
+ <g id="shape129-508" v:mID="129" v:groupContext="shape" transform="translate(7.78787,-4.53251)">
+ <title>Sheet.129</title>
+ <path d="M9.08 334.57 L9.03 342.55 L7.25 341.57 L9.08 334.57 ZM6.66 333.14 L6.61 341.11 L4.83 340.13 L6.66
+ 333.14 ZM4.25 331.7 L4.2 339.68 L2.42 338.7 L4.25 331.7 ZM1.83 330.27 L1.78 338.24 L0 337.27
+ L1.83 330.27 Z" class="st42"/>
+ <path d="M9.08 334.57 L9.03 342.55 L7.25 341.57M6.66 333.14 L6.61 341.11 L4.83 340.13M4.25 331.7 L4.2 339.68
+ L2.42 338.7M1.83 330.27 L1.78 338.24 L0 337.27" class="st44"/>
+ </g>
+ <g id="shape130-511" v:mID="130" v:groupContext="shape" transform="translate(2.22125,-11.8454)">
+ <title>Sheet.130</title>
+ <path d="M0 341.85 L0.63 341.42 L1.42 341.78 L0.03 342.55 L0 341.85 Z" class="st45"/>
+ </g>
+ <g id="shape131-514" v:mID="131" v:groupContext="shape" transform="translate(17.1796,-3.17487)">
+ <title>Sheet.131</title>
+ <path d="M0 341.85 L0.63 341.42 L1.42 341.78 L0.03 342.55 L0 341.85 Z" class="st45"/>
+ </g>
+ <g id="shape132-517" v:mID="132" v:groupContext="shape" transform="translate(1.46036,-10.3893)">
+ <title>Sheet.132</title>
+ <path d="M2.12 341.35 L0 342.55 L0 333.54 L2.12 332.29 L2.12 333.41 L0.79 334.15 L0.79 341.09 L2.18 340.33
+ L2.12 341.35 Z" class="st36"/>
+ </g>
+ <g id="shape133-520" v:mID="133" v:groupContext="shape" transform="translate(16.4187,-1.71875)">
+ <title>Sheet.133</title>
+ <path d="M2.12 341.35 L0 342.55 L0 333.54 L2.12 332.29 L2.12 333.41 L0.79 334.15 L0.79 341.09 L2.18 340.33
+ L2.12 341.35 Z" class="st36"/>
+ </g>
+ <g id="shape134-523" v:mID="134" v:groupContext="shape" transform="translate(0.467548,-10.3893)">
+ <title>Sheet.134</title>
+ <path d="M0.99 333.54 L3.11 332.29 L2.12 331.66 L0 332.91 L0 341.92 L0.99 342.55 L0.99 333.54 Z"
+ class="st46"/>
+ </g>
+ <g id="shape135-526" v:mID="135" v:groupContext="shape" transform="translate(15.4259,-1.71875)">
+ <title>Sheet.135</title>
+ <path d="M0.99 333.54 L3.11 332.29 L2.12 331.66 L0 332.91 L0 341.92 L0.99 342.55 L0.99 333.54 Z"
+ class="st46"/>
+ </g>
+ <g id="shape136-529" v:mID="136" v:groupContext="shape" transform="translate(0.467548,-1.71928)">
+ <title>Sheet.136</title>
+ <path d="M17.34 339.96 L16.75 340.37 L16.75 334.15 L18.07 333.41 L18.07 332.29 L17.08 331.66 L14.96 332.91
+ L14.96 341.92 L15.95 342.55 L18.07 341.35 L18.14 340.33 L17.34 339.96 ZM2.38 331.29 L1.79 331.7
+ L1.79 325.48 L3.11 324.74 L3.11 323.62 L2.12 322.99 L0 324.24 L0 333.25 L0.99 333.87 L3.11 332.68
+ L3.18 331.66 L2.38 331.29 Z" class="st47"/>
+ </g>
+ <g id="shape137-531" v:mID="137" v:groupContext="shape" transform="translate(19.9526,-8.71396)">
+ <title>Sheet.137</title>
+ <path d="M1.13 341.58 a0.653986 0.653986 -180 0 0 -0.73971 -0.492434 a0.656072 0.656072 -180 0 0 -0.253101
+ 0.865731 a0.653066 0.653066 -180 0 0 0.740769 0.491375 a0.655459 0.655459 -180 0 0 0.252042
+ -0.864672 Z" class="st38"/>
+ </g>
+ <g id="shape138-534" v:mID="138" v:groupContext="shape" transform="translate(19.9526,-2.35997)">
+ <title>Sheet.138</title>
+ <path d="M1.13 341.58 a0.653986 0.653986 -180 0 0 -0.73971 -0.492434 a0.656072 0.656072 -180 0 0 -0.253101
+ 0.865731 a0.653066 0.653066 -180 0 0 0.740769 0.491375 a0.655459 0.655459 -180 0 0 0.252042
+ -0.864672 Z" class="st38"/>
+ </g>
+ <g id="shape139-537" v:mID="139" v:groupContext="shape" transform="translate(0,-0.415652)">
+ <title>Sheet.139</title>
+ <path d="M36.07 331.28 L36.07 321.27 L19.64 311.85 L3.16 321.13 L3.16 321.52 L0 319.68 L0 329.58 L0.47 329.86
+ L0.47 331.94 L1.46 332.57 L3.33 331.52 L15.43 338.57 L15.43 340.61 L16.42 341.24 L18.24 340.22
+ L22.24 342.55 L22.24 339.27 L36.07 331.28 Z" class="st47"/>
+ </g>
+ <g id="shape140-539" v:mID="140" v:groupContext="shape" transform="translate(27.8077,-2.86477)">
+ <title>Sheet.140</title>
+ <path d="M0.29 342.55 L6.62 338.89 A1.82805 1.82805 0 0 1 6.62 336.9 L0.29 340.55 A1.82805 1.82805 -180 0
+ 0 0.29 342.55 Z" class="st48"/>
+ </g>
+ <g id="shape141-541" v:mID="141" v:groupContext="shape" transform="translate(23.5035,-4.85627)">
+ <title>Sheet.141</title>
+ <path d="M4.6 342.55 L10.92 338.89 L6.32 336.24 L0 339.89 L4.6 342.55 Z" class="st49"/>
+ </g>
+ <g id="shape142-544" v:mID="142" v:groupContext="shape" transform="translate(23.3588,-2.86477)">
+ <title>Sheet.142</title>
+ <path d="M0.14 339.89 L4.74 342.55 A1.82805 1.82805 0 0 1 4.74 340.55 L0.14 337.9 A3.49826 3.49826 -180 0
+ 0 0.14 339.89 Z" class="st50"/>
+ </g>
+ <g id="shape143-547" v:mID="143" v:groupContext="shape" transform="translate(25.8933,-5.98478)">
+ <title>Sheet.143</title>
+ <path d="M2.87 342.55 L0 340.89" class="st51"/>
+ <path d="M0.94 340.34 L3.82 342" class="st51"/>
+ <path d="M1.88 339.8 L4.76 341.46" class="st51"/>
+ <path d="M2.82 339.26 L5.7 340.92" class="st51"/>
+ <path d="M3.76 338.71 L6.64 340.37" class="st51"/>
+ </g>
+ <g id="shape144-554" v:mID="144" v:groupContext="shape" transform="translate(23.5035,-7.51159)">
+ <title>Sheet.144</title>
+ <path d="M5.13 341.17 L11.45 337.52 A11.9345 11.9345 0 0 1 6.32 338.89 L0 342.55 A11.9345 11.9345 -180 0
+ 0 5.13 341.17 Z" class="st52"/>
+ </g>
+ <g id="shape145-557" v:mID="145" v:groupContext="shape" transform="translate(30.2106,-4.74563)">
+ <title>Sheet.145</title>
+ <path d="M0.98 341.98 L0 342.55" class="st51"/>
+ <path d="M1.26 341.48 L2.24 340.92" class="st51"/>
+ <path d="M2.53 340.42 L3.51 339.86" class="st51"/>
+ </g>
+ <g id="shape146-562" v:mID="146" v:groupContext="shape" transform="translate(23.3588,-2.86477)">
+ <title>Sheet.146</title>
+ <path d="M0.14 339.89 L4.74 342.55 L11.07 338.89 A1.82805 1.82805 0 0 1 11.07 336.9 L7.85 335.04 L11.6 332.87
+ A11.9345 11.9345 0 0 1 6.47 334.25 L0.14 337.9 A3.49826 3.49826 -180 0 0 0.14 339.89"
+ class="st53"/>
+ </g>
+ </g>
+ <g id="shape147-565" v:mID="147" v:groupContext="shape" transform="translate(427.321,214.49) rotate(90)">
+ <title>Cloud</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <path d="M5.37 311.54 A8.61618 10.0654 0 0 1 9.5 292.2 A17.4727 20.4114 0 0 1 34.86 275.89 A20.0634 23.4379 0
+ 0 1 56.58 272.26 A12.5816 14.6977 0 0 1 75.21 271.05 A14.3244 16.7336 0 0 1 97.98 277.09 A10.2423
+ 11.9646 0 0 1 106.25 294.02 A12.6864 14.8197 0 0 1 95.9 318.19 A16.0049 18.6962 0 0 1 73.14 330.27
+ A18.8712 22.0444 0 0 1 42.1 335.11 A23.9217 27.9441 0 0 1 15.2 330.27 A9.43759 11.0249 0 0 1 5.37
+ 311.54 Z" class="st42"/>
+ <path d="M5.37 311.54 A8.61618 10.0654 0 0 1 9.5 292.2 A17.4727 20.4114 0 0 1 34.86 275.89 A20.0634 23.4379 0
+ 0 1 56.58 272.26 A12.5816 14.6977 0 0 1 75.21 271.05 A14.3244 16.7336 0 0 1 97.98 277.09 A10.2423
+ 11.9646 0 0 1 106.25 294.02 A12.6864 14.8197 0 0 1 95.9 318.19 A16.0049 18.6962 0 0 1 73.14 330.27
+ A18.8712 22.0444 0 0 1 42.1 335.11 A23.9217 27.9441 0 0 1 15.2 330.27 A9.43759 11.0249 0 0 1 5.37
+ 311.54" class="st54"/>
+ <path d="M11.05 312.14 A8.59237 10.0375 0 0 1 5.37 311.54" class="st54"/>
+ <path d="M40.54 332.09 A8.62978 10.0812 -180 0 0 42.1 335.11" class="st54"/>
+ <path d="M73.92 326.65 A6.96633 8.13801 0 0 1 73.14 330.27" class="st54"/>
+ <path d="M89.7 308.52 A7.30994 8.5394 0 0 1 95.9 318.19" class="st54"/>
+ <path d="M103.15 297.64 A6.67364 7.79609 -180 0 0 106.25 294.02" class="st54"/>
+ <path d="M37.96 278.3 A10.2914 12.0219 -180 0 0 34.86 275.89" class="st54"/>
+ </g>
+ <g id="shape148-574" v:mID="148" v:groupContext="shape" transform="translate(110.222,-64.9346)">
+ <title>Triangle</title>
+ <desc>setsum</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="15.6995" cy="336.449" width="31.4" height="12.1933"/>
+ <g id="shadow148-575" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <path d="M31.4 342.55 L14.67 324.26 L0 342.55 L31.4 342.55 Z" class="st9"/>
+ </g>
+ <path d="M31.4 342.55 L14.67 324.26 L0 342.55 L31.4 342.55 Z" class="st10"/>
+ <text x="8.35" y="337.95" class="st55" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>setsum</text> </g>
+ <g id="shape149-579" v:mID="149" v:groupContext="shape" transform="translate(292.639,20.8827) rotate(45)">
+ <title>Sheet.149</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L14.71 342.55" class="st14"/>
+ </g>
+ <g id="shape150-584" v:mID="150" v:groupContext="shape" transform="translate(43.2897,-54.1122)">
+ <title>Sheet.150</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L10.07 342.55" class="st14"/>
+ </g>
+ <g id="shape151-589" v:mID="151" v:groupContext="shape" transform="translate(-112.261,8.34531) rotate(-28.1394)">
+ <title>Sheet.151</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 342.55 L18 342.55" class="st14"/>
+ </g>
+ <g id="shape152-594" v:mID="152" v:groupContext="shape">
+ <title>Sheet.152</title>
+ <desc>Clients</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="32.4673" cy="337.134" width="64.94" height="10.8224"/>
+ <rect x="0" y="331.723" width="64.9346" height="10.8224" class="st11"/>
+ <text x="21.5" y="339.53" class="st6" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Clients</text> </g>
+ <g id="shape153-597" v:mID="153" v:groupContext="shape" transform="translate(83.578,-9.58078)">
+ <title>Sheet.153</title>
+ <desc>Distributed Cache</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="42.0677" cy="337.134" width="84.14" height="10.8224"/>
+ <rect x="0" y="331.723" width="84.1355" height="10.8224" class="st11"/>
+ <text x="13.1" y="339.53" class="st6" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Distributed Cache</text> </g>
+ <g id="shape154-600" v:mID="154" v:groupContext="shape" transform="translate(181.983,-18.6146)">
+ <title>Sheet.154</title>
+ <desc>Web Servers</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="32.4673" cy="337.134" width="64.94" height="10.8224"/>
+ <rect x="0" y="331.723" width="64.9346" height="10.8224" class="st11"/>
+ <text x="11.93" y="339.53" class="st6" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Web Servers</text> </g>
+ <g id="shape155-603" v:mID="155" v:groupContext="shape" transform="translate(96.6068,630.978) rotate(180)">
+ <title>Simple Arrow</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="ArrowType" v:prompt="" v:val="VT0(2):26"/>
+ </v:userDefs>
+ <g id="shadow155-604" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,-0.3456,-1.9728)" class="st1">
+ <path d="M0 342.55 L12 330.55 L12 336.55 L16.23 336.55 L16.23 342.55 L16.23 348.54 L12 348.54 L12 354.54
+ L0 342.55 Z" class="st56"/>
+ </g>
+ <path d="M0 342.55 L12 330.55 L12 336.55 L16.23 336.55 L16.23 342.55 L16.23 348.54 L12 348.54 L12 354.54 L0 342.55
+ Z" class="st57"/>
+ </g>
+ <g id="shape156-607" v:mID="156" v:groupContext="shape" transform="translate(173.159,625.567) rotate(180)">
+ <title>Simple Arrow.153</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="ArrowType" v:prompt="" v:val="VT0(2):26"/>
+ </v:userDefs>
+ <g id="shadow156-608" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,-0.3456,-1.9728)" class="st1">
+ <path d="M0 342.55 L12 330.55 L12 336.55 L16.23 336.55 L16.23 342.55 L16.23 348.54 L12 348.54 L12 354.54
+ L0 342.55 Z" class="st56"/>
+ </g>
+ <path d="M0 342.55 L12 330.55 L12 336.55 L16.23 336.55 L16.23 342.55 L16.23 348.54 L12 348.54 L12 354.54 L0 342.55
+ Z" class="st57"/>
+ </g>
+ </g>
+ <g id="shape157-611" v:mID="157" v:groupContext="shape" transform="translate(0,-149.475)">
+ <title>Rectangle</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow157-612" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="193.823" width="271.116" height="148.722" class="st58"/>
+ </g>
+ <rect x="0" y="193.823" width="271.116" height="148.722" class="st59"/>
+ </g>
+ <g id="shape158-615" v:mID="158" v:groupContext="shape" transform="translate(271.116,-149.475)">
+ <title>Rectangle.158</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow158-616" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="193.823" width="202.104" height="148.722" class="st58"/>
+ </g>
+ <rect x="0" y="193.823" width="202.104" height="148.722" class="st59"/>
+ </g>
+ <g id="shape159-619" v:mID="159" v:groupContext="shape">
+ <title>Rectangle.159</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow159-620" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="193.823" width="271.116" height="148.722" class="st58"/>
+ </g>
+ <rect x="0" y="193.823" width="271.116" height="148.722" class="st59"/>
+ </g>
+ <g id="shape160-623" v:mID="160" v:groupContext="shape" transform="translate(271.116,0)">
+ <title>Rectangle.160</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow160-624" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="193.823" width="202.104" height="148.722" class="st58"/>
+ </g>
+ <rect x="0" y="193.823" width="202.104" height="148.722" class="st59"/>
+ </g>
+ <g id="shape161-627" v:mID="161" v:groupContext="shape" transform="translate(83.578,-151.241)">
+ <title>Sheet.161</title>
+ <desc>(a) Distributed Web Cache</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54.3546" cy="333.806" width="108.71" height="17.4792"/>
+ <g id="shadow161-628" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="325.066" width="108.709" height="17.4792" class="st9"/>
+ </g>
+ <rect x="0" y="325.066" width="108.709" height="17.4792" class="st10"/>
+ <text x="4" y="336.81" class="st60" v:langID="1033"><v:paragraph v:spLine="-1.5" v:spBefore="8" v:spAfter="16"
+ v:bulletSize="0.166667"/><v:tabList/>(a) Distributed Web Cache</text> </g>
+ <g id="shape162-632" v:mID="162" v:groupContext="shape" transform="translate(319.513,-151.241)">
+ <title>Sheet.162</title>
+ <desc>(b) Detecting Routing Loops</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54.3546" cy="333.806" width="108.71" height="17.4792"/>
+ <g id="shadow162-633" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="325.066" width="108.709" height="17.4792" class="st9"/>
+ </g>
+ <rect x="0" y="325.066" width="108.709" height="17.4792" class="st10"/>
+ <text x="4" y="336.81" class="st60" v:langID="1033"><v:paragraph v:spLine="-1.5" v:spBefore="8" v:spAfter="16"
+ v:bulletSize="0.166667"/><v:tabList/>(b) Detecting Routing Loops</text> </g>
+ <g id="shape163-637" v:mID="163" v:groupContext="shape" transform="translate(77.5283,-3.35965)">
+ <title>Sheet.163</title>
+ <desc>(c) In-order Workload Scheduler</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="63.5211" cy="333.806" width="127.05" height="17.4792"/>
+ <g id="shadow163-638" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="325.066" width="127.042" height="17.4792" class="st9"/>
+ </g>
+ <rect x="0" y="325.066" width="127.042" height="17.4792" class="st10"/>
+ <text x="4" y="336.81" class="st60" v:langID="1033"><v:paragraph v:spLine="-1.5" v:spBefore="8" v:spAfter="16"
+ v:bulletSize="0.166667"/><v:tabList/>(c) In-order Workload Scheduler</text> </g>
+ <g id="shape164-642" v:mID="164" v:groupContext="shape" transform="translate(307.414,-3.35965)">
+ <title>Sheet.164</title>
+ <desc>(d) Database Semi-join Operations</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="66.2253" cy="333.806" width="132.46" height="17.4792"/>
+ <g id="shadow164-643" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="325.066" width="132.451" height="17.4792" class="st9"/>
+ </g>
+ <rect x="0" y="325.066" width="132.451" height="17.4792" class="st10"/>
+ <text x="4" y="336.81" class="st60" v:langID="1033"><v:paragraph v:spLine="-1.5" v:spBefore="8" v:spAfter="16"
+ v:bulletSize="0.166667"/><v:tabList/>(d) Database Semi-join Operations</text> </g>
+ </g>
+ </g>
+</svg>
diff --git a/doc/guides/prog_guide/img/member_i2.svg b/doc/guides/prog_guide/img/member_i2.svg
new file mode 100644
index 0000000..759c654
--- /dev/null
+++ b/doc/guides/prog_guide/img/member_i2.svg
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by Microsoft Visio, SVG Export memship_i2.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+ xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="4.38194in" height="1.25694in"
+ viewBox="0 0 315.5 90.5" xml:space="preserve" color-interpolation-filters="sRGB" class="st6">
+ <v:documentProperties v:langID="1033" v:viewMarkup="false"/>
+
+ <style type="text/css">
+ <![CDATA[
+ .st1 {fill:none;stroke:none;stroke-width:0.25}
+ .st2 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em}
+ .st3 {baseline-shift:32.4943%;font-size:0.649886em}
+ .st4 {font-size:1em}
+ .st5 {font-family:Cambria Math;font-size:1em}
+ .st6 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+ ]]>
+ </style>
+
+ <g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+ <title>Page-1</title>
+ <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+ <g id="shape3-1" v:mID="3" v:groupContext="shape" transform="translate(0.25,-0.25)">
+ <title>Sheet.3</title>
+ <desc>False Positive Probability = (1-(1-1/m)kn)k ≃ (1-ekn/m)k</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="157.5" cy="45.5" width="315" height="90"/>
+ <rect x="0" y="0.5" width="315" height="90" class="st1"/>
+ <text x="8.28" y="49.82" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>False Positive Probability = (1-(1-1/m)<tspan
+ dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan dy="0.152em" class="st4">)</tspan><tspan
+ dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan><tspan dy="0.152em" class="st4"> </tspan><tspan
+ class="st5">≃</tspan> (1-e<tspan dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan class="st3"
+ v:baseFontSize="14">/</tspan><tspan class="st3" v:baseFontSize="14">m</tspan><tspan dy="0.152em"
+ class="st4">)</tspan><tspan dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan></text> </g>
+ </g>
+</svg>
diff --git a/doc/guides/prog_guide/img/member_i3.svg b/doc/guides/prog_guide/img/member_i3.svg
new file mode 100644
index 0000000..41e92cb
--- /dev/null
+++ b/doc/guides/prog_guide/img/member_i3.svg
@@ -0,0 +1,148 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by Microsoft Visio, SVG Export memship_i3.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+ width="4.71875in" height="2.84375in" viewBox="0 0 339.75 204.75" xml:space="preserve" color-interpolation-filters="sRGB"
+ class="st14">
+ <style type="text/css">
+ <![CDATA[
+ .st1 {visibility:visible}
+ .st2 {fill:#5b9bd5;fill-opacity:0.22;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.22}
+ .st3 {fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25}
+ .st4 {marker-end:url(#mrkr5-32);stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st5 {fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-opacity:1;stroke-width:0.28409090909091}
+ .st6 {fill:#feffff;font-family:Calibri;font-size:0.833336em}
+ .st7 {font-size:1em}
+ .st8 {fill:#deebf6;stroke:#c7c8c8;stroke-width:0.25}
+ .st9 {fill:#000000;font-family:Calibri;font-size:0.833336em}
+ .st10 {marker-end:url(#mrkr5-84);stroke:#ff0000;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st11 {fill:#ff0000;fill-opacity:1;stroke:#ff0000;stroke-opacity:1;stroke-width:0.28409090909091}
+ .st12 {fill:none;stroke:none;stroke-width:0.25}
+ .st13 {fill:#ff0000;font-family:Calibri;font-size:1.00001em}
+ .st14 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+ ]]>
+ </style>
+
+ <defs id="Markers">
+ <g id="lend5">
+ <path d="M 2 1 L 0 0 L 1.98117 -0.993387 C 1.67173 -0.364515 1.67301 0.372641 1.98465 1.00043 " style="stroke:none"/>
+ </g>
+ <marker id="mrkr5-32" class="st5" refX="-6.16" orient="auto" markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-3.52,-3.52) "/>
+ </marker>
+ <marker id="mrkr5-84" class="st11" refX="-5.8" orient="auto" markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-3.52,-3.52) "/>
+ </marker>
+ </defs>
+ <defs id="Filters">
+ <filter id="filter_2">
+ <feGaussianBlur stdDeviation="2"/>
+ </filter>
+ </defs>
+ <g>
+ <title>Page-1</title>
+ <g id="group174-1" transform="translate(3.0294,-5.3478)">
+ <title>Sheet.174</title>
+ <g id="shape155-2" transform="translate(99,-99)">
+ <title>Circle</title>
+ <g id="shadow155-3" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st2"/>
+ </g>
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st3"/>
+ </g>
+ <g id="shape156-7" transform="translate(207,-108)">
+ <title>Circle.156</title>
+ <g id="shadow156-8" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st2"/>
+ </g>
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st3"/>
+ </g>
+ <g id="shape157-12" transform="translate(153,-162)">
+ <title>Circle.157</title>
+ <g id="shadow157-13" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st2"/>
+ </g>
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st3"/>
+ </g>
+ <g id="shape158-17" transform="translate(297,-99)">
+ <title>Circle.158</title>
+ <g id="shadow158-18" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st2"/>
+ </g>
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st3"/>
+ </g>
+ <g id="shape159-22" transform="translate(180,-36)">
+ <title>Circle.159</title>
+ <g id="shadow159-23" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st2"/>
+ </g>
+ <path d="M0 186.75 A18 18 0 0 1 36 186.75 A18 18 0 1 1 0 186.75 Z" class="st3"/>
+ </g>
+ <g id="shape160-27" transform="translate(109.604,-115.419) rotate(-7.12502)">
+ <title>Sheet.160</title>
+ <path d="M0 204.75 L66.4 204.75" class="st4"/>
+ </g>
+ <g id="shape161-33" transform="translate(276.661,-123.214) rotate(9.46232)">
+ <title>Sheet.161</title>
+ <path d="M0 204.75 L48.58 204.75" class="st4"/>
+ </g>
+ <g id="shape162-38" transform="translate(246.135,262.572) rotate(-160.346)">
+ <title>Sheet.162</title>
+ <path d="M0 204.75 L127.63 204.75" class="st4"/>
+ </g>
+ <g id="shape163-43" transform="translate(284.391,198.775) rotate(141.977)">
+ <title>Sheet.163</title>
+ <path d="M0 204.75 L46.23 204.75" class="st4"/>
+ </g>
+ <g id="shape164-48" transform="translate(70.6118,307.655) rotate(-145.945)">
+ <title>Sheet.164</title>
+ <path d="M0 204.75 L60.88 204.75" class="st4"/>
+ </g>
+ <g id="shape167-53" transform="translate(72,0)">
+ <title>Rectangle.167</title>
+ <desc>BF of IDs</desc>
+ <g id="shadow167-54" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <rect x="0" y="177.75" width="36" height="27" class="st2"/>
+ </g>
+ <rect x="0" y="177.75" width="36" height="27" class="st3"/>
+ <text x="7.69" y="188.25" class="st6">BF of <tspan x="11.71" dy="1.2em" class="st7">IDs</tspan></text> </g>
+ <g id="shape168-60" transform="translate(108,0)">
+ <title>Rectangle.168</title>
+ <desc>Packet</desc>
+ <g id="shadow168-61" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <rect x="0" y="177.75" width="90" height="27" class="st2"/>
+ </g>
+ <rect x="0" y="177.75" width="90" height="27" class="st8"/>
+ <text x="31.47" y="194.25" class="st9">Packet</text> </g>
+ <g id="shape169-66" transform="translate(0,-63)">
+ <title>Rectangle.169</title>
+ <desc>BF of IDs</desc>
+ <g id="shadow169-67" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <rect x="0" y="177.75" width="36" height="27" class="st2"/>
+ </g>
+ <rect x="0" y="177.75" width="36" height="27" class="st3"/>
+ <text x="7.69" y="188.25" class="st6">BF of <tspan x="11.71" dy="1.2em" class="st7">IDs</tspan></text> </g>
+ <g id="shape170-73" transform="translate(36,-63)">
+ <title>Rectangle.170</title>
+ <desc>Packet</desc>
+ <g id="shadow170-74" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1">
+ <rect x="0" y="177.75" width="90" height="27" class="st2"/>
+ </g>
+ <rect x="0" y="177.75" width="90" height="27" class="st8"/>
+ <text x="31.47" y="194.25" class="st9">Packet</text> </g>
+ <g id="shape171-79" transform="translate(240.248,331.493) rotate(161.565)">
+ <title>Sheet.171</title>
+ <path d="M-0 190.52 A81.3416 36.0611 -153.48 0 0 82.31 195.86 L82.49 195.55" class="st10"/>
+ </g>
+ <g id="shape172-85" transform="translate(156.426,260.029) rotate(161.565)">
+ <title>Sheet.172</title>
+ <path d="M-0 181.6 A88.1422 54.1439 -124.1 0 0 82.68 187.13 L82.83 186.81" class="st10"/>
+ </g>
+ <g id="shape173-90" transform="translate(18,-121.5)">
+ <title>Sheet.173</title>
+ <desc>Encode ID</desc>
+ <rect x="0" y="177.75" width="63" height="27" class="st12"/>
+ <text x="7.02" y="194.85" class="st13">Encode ID</text> </g>
+ </g>
+ </g>
+</svg>
diff --git a/doc/guides/prog_guide/img/member_i4.svg b/doc/guides/prog_guide/img/member_i4.svg
new file mode 100644
index 0000000..a2b6f2f
--- /dev/null
+++ b/doc/guides/prog_guide/img/member_i4.svg
@@ -0,0 +1,450 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.0//EN" "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd">
+<!-- Generated by Microsoft Visio 11.0, SVG Export, v1.0 memship_i4.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+ xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="7.625in" height="3.125in" viewBox="0 0 549 225"
+ xml:space="preserve" color-interpolation-filters="sRGB" class="st18">
+ <v:documentProperties v:langID="1033" v:viewMarkup="false"/>
+
+ <style type="text/css">
+ <![CDATA[
+ .st1 {fill:none;stroke:#ff0000;stroke-width:0.25}
+ .st2 {fill:#5b9bd5;font-family:Calibri;font-size:1.00001em}
+ .st3 {marker-end:url(#mrkr5-10);stroke:#ff0000;stroke-linecap:round;stroke-linejoin:round;stroke-width:2.25}
+ .st4 {fill:#ff0000;fill-opacity:1;stroke:#ff0000;stroke-opacity:1;stroke-width:0.47169811320755}
+ .st5 {visibility:visible}
+ .st6 {fill:#5b9bd5;fill-opacity:0.22;stroke:#5b9bd5;stroke-opacity:0.22;stroke-width:0.25}
+ .st7 {fill:#5b9bd5;stroke:#c8c8c8;stroke-width:0.25}
+ .st8 {fill:none;stroke:none;stroke-width:0.25}
+ .st9 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em}
+ .st10 {stroke:#5b9bd5;stroke-dasharray:1.5,3;stroke-linecap:round;stroke-linejoin:round;stroke-width:1.5}
+ .st11 {fill:#5b9bd5;fill-opacity:0.25;stroke:#5b9bd5;stroke-opacity:0.25;stroke-width:0.75}
+ .st12 {fill:#4f88bb;stroke:#41719c;stroke-width:0.75}
+ .st13 {fill:#ffffff;font-family:Calibri;font-size:1.00001em;font-weight:bold}
+ .st14 {fill:#000000;font-family:Calibri;font-size:1.00001em}
+ .st15 {font-size:1em}
+ .st16 {marker-end:url(#mrkr5-162);stroke:#000000;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st17 {fill:#000000;fill-opacity:1;stroke:#000000;stroke-opacity:1;stroke-width:0.28409090909091}
+ .st18 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+ ]]>
+ </style>
+
+ <defs id="Markers">
+ <g id="lend5">
+ <path d="M 2 1 L 0 0 L 1.98117 -0.993387 C 1.67173 -0.364515 1.67301 0.372641 1.98465 1.00043 " style="stroke:none"/>
+ </g>
+ <marker id="mrkr5-10" class="st4" v:arrowType="5" v:arrowSize="2" v:setback="3.71" refX="-3.71" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-2.12,-2.12) "/>
+ </marker>
+ <marker id="mrkr5-162" class="st17" v:arrowType="5" v:arrowSize="2" v:setback="5.8" refX="-5.8" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-3.52,-3.52) "/>
+ </marker>
+ </defs>
+ <g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+ <title>Page-1</title>
+ <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+ <g id="group47-1" transform="translate(3.0294,-0.25)" v:mID="47" v:groupContext="group">
+ <title>Sheet.47</title>
+ <g id="shape1-2" v:mID="1" v:groupContext="shape" transform="translate(177.75,-191.922)">
+ <title>Sheet.1</title>
+ <desc>Element</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="216" width="108" height="18"/>
+ <rect x="0" y="207" width="108" height="18" class="st1"/>
+ <text x="33.77" y="219.6" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Element</text> </g>
+ <g id="shape2-5" v:mID="2" v:groupContext="shape" transform="translate(456.75,33.0781) rotate(90)">
+ <title>Sheet.2</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 225 L18.65 225" class="st3"/>
+ </g>
+ <g id="shape3-11" v:mID="3" v:groupContext="shape" transform="translate(0,-67.0469)">
+ <title>Rectangle.54</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow3-12" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape4-15" v:mID="4" v:groupContext="shape" transform="translate(27,-67.0469)">
+ <title>Rectangle.55</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow4-16" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape5-19" v:mID="5" v:groupContext="shape" transform="translate(54,-67.0469)">
+ <title>Rectangle.56</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow5-20" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape6-23" v:mID="6" v:groupContext="shape" transform="translate(0,-53.5469)">
+ <title>Rectangle.57</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow6-24" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape7-27" v:mID="7" v:groupContext="shape" transform="translate(27,-53.5469)">
+ <title>Rectangle.58</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow7-28" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape8-31" v:mID="8" v:groupContext="shape" transform="translate(54,-53.5469)">
+ <title>Rectangle.59</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow8-32" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape9-35" v:mID="9" v:groupContext="shape" transform="translate(5.625,-72.6719)">
+ <title>Sheet.9</title>
+ <desc>BF-1</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="36" cy="211.5" width="72" height="27"/>
+ <rect x="0" y="198" width="72" height="27" class="st8"/>
+ <text x="23.29" y="215.7" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>BF-1</text> </g>
+ <g id="shape10-38" v:mID="10" v:groupContext="shape" transform="translate(128.25,-65.0781)">
+ <title>Rectangle.74</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow10-39" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape11-42" v:mID="11" v:groupContext="shape" transform="translate(155.25,-65.0781)">
+ <title>Rectangle.75</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow11-43" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape12-46" v:mID="12" v:groupContext="shape" transform="translate(182.25,-65.0781)">
+ <title>Rectangle.76</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow12-47" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape13-50" v:mID="13" v:groupContext="shape" transform="translate(128.25,-51.5781)">
+ <title>Rectangle.77</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow13-51" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape14-54" v:mID="14" v:groupContext="shape" transform="translate(155.25,-51.5781)">
+ <title>Rectangle.78</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow14-55" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape15-58" v:mID="15" v:groupContext="shape" transform="translate(182.25,-51.5781)">
+ <title>Rectangle.79</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow15-59" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape16-62" v:mID="16" v:groupContext="shape" transform="translate(301.5,-65.0781)">
+ <title>Rectangle.81</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow16-63" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape17-66" v:mID="17" v:groupContext="shape" transform="translate(328.5,-65.0781)">
+ <title>Rectangle.82</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow17-67" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape18-70" v:mID="18" v:groupContext="shape" transform="translate(355.5,-65.0781)">
+ <title>Rectangle.83</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow18-71" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape19-74" v:mID="19" v:groupContext="shape" transform="translate(301.5,-51.5781)">
+ <title>Rectangle.84</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow19-75" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape20-78" v:mID="20" v:groupContext="shape" transform="translate(328.5,-51.5781)">
+ <title>Rectangle.85</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow20-79" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape21-82" v:mID="21" v:groupContext="shape" transform="translate(355.5,-51.5781)">
+ <title>Rectangle.86</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow21-83" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape22-86" v:mID="22" v:groupContext="shape" transform="translate(447.75,-65.6406)">
+ <title>Rectangle.88</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow22-87" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape23-90" v:mID="23" v:groupContext="shape" transform="translate(474.75,-65.6406)">
+ <title>Rectangle.89</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow23-91" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape24-94" v:mID="24" v:groupContext="shape" transform="translate(501.75,-65.6406)">
+ <title>Rectangle.90</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow24-95" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape25-98" v:mID="25" v:groupContext="shape" transform="translate(447.75,-52.1406)">
+ <title>Rectangle.91</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow25-99" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape26-102" v:mID="26" v:groupContext="shape" transform="translate(474.75,-52.1406)">
+ <title>Rectangle.92</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow26-103" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape27-106" v:mID="27" v:groupContext="shape" transform="translate(501.75,-52.1406)">
+ <title>Rectangle.93</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow27-107" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st5">
+ <rect x="0" y="211.5" width="27" height="13.5" class="st6"/>
+ </g>
+ <rect x="0" y="211.5" width="27" height="13.5" class="st7"/>
+ </g>
+ <g id="shape28-110" v:mID="28" v:groupContext="shape" transform="translate(213.75,-63.9531)">
+ <title>Sheet.28</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 225 L83.25 225" class="st10"/>
+ </g>
+ <g id="shape29-113" v:mID="29" v:groupContext="shape" transform="translate(387,-63.9531)">
+ <title>Sheet.29</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 225 L54 225" class="st10"/>
+ </g>
+ <g id="group31-116" transform="translate(184.5,-113.172)" v:mID="31" v:groupContext="group">
+ <title>Sheet.31</title>
+ <g id="shape32-117" v:mID="32" v:groupContext="shape" transform="translate(225,173.25) rotate(90)">
+ <title>Block Arrow</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow32-118" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,1.9728,-0.3456)" class="st5">
+ <path d="M0 225 L25.87 225 L51.75 177.75 L25.87 130.5 L0 130.5 L0 225 Z" class="st11"/>
+ </g>
+ <path d="M0 225 L25.87 225 L51.75 177.75 L25.87 130.5 L0 130.5 L0 225 Z" class="st12"/>
+ </g>
+ <g id="shape33-121" v:mID="33" v:groupContext="shape" transform="translate(2.25,-24.3529)">
+ <title>Sheet.33</title>
+ <desc>h1, h2 .. hk</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="45" cy="215.868" width="90" height="18.2647"/>
+ <rect x="0" y="206.735" width="90" height="18.2647" class="st8"/>
+ <text x="17.56" y="219.47" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>h1, h2 .. hk</text> </g>
+ </g>
+ <g id="shape34-124" v:mID="34" v:groupContext="shape" transform="translate(307.011,286.73) rotate(152.323)">
+ <title>Sheet.34</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 225 L128.85 225" class="st3"/>
+ </g>
+ <g id="shape35-129" v:mID="35" v:groupContext="shape" transform="translate(433.272,125.452) rotate(99.7172)">
+ <title>Sheet.35</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 225 L58.31 225" class="st3"/>
+ </g>
+ <g id="shape36-134" v:mID="36" v:groupContext="shape" transform="translate(407.724,-64.1459) rotate(45)">
+ <title>Sheet.36</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 225 L79.16 225" class="st3"/>
+ </g>
+ <g id="shape37-139" v:mID="37" v:groupContext="shape" transform="translate(320.441,-127.12) rotate(15.6155)">
+ <title>Sheet.37</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 225 L200.75 225" class="st3"/>
+ </g>
+ <g id="shape38-144" v:mID="38" v:groupContext="shape" transform="translate(132.75,-75.2588)">
+ <title>Sheet.38</title>
+ <desc>BF-2</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="36" cy="211.5" width="72" height="27"/>
+ <rect x="0" y="198" width="72" height="27" class="st8"/>
+ <text x="23.29" y="215.7" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>BF-2</text> </g>
+ <g id="shape39-147" v:mID="39" v:groupContext="shape" transform="translate(303.75,-70.7588)">
+ <title>Sheet.39</title>
+ <desc>BF-X</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="36" cy="211.5" width="72" height="27"/>
+ <rect x="0" y="198" width="72" height="27" class="st8"/>
+ <text x="23.2" y="215.7" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>BF-X</text> </g>
+ <g id="shape40-150" v:mID="40" v:groupContext="shape" transform="translate(447.75,-75.2588)">
+ <title>Sheet.40</title>
+ <desc>BF-L</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="36" cy="211.5" width="72" height="27"/>
+ <rect x="0" y="198" width="72" height="27" class="st8"/>
+ <text x="23.89" y="215.7" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>BF-L</text> </g>
+ <g id="shape41-153" v:mID="41" v:groupContext="shape" transform="translate(300.375,-117)">
+ <title>Sheet.41</title>
+ <desc>Hashing for lookup/Insertion into a vector of BFs happens once</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="90" cy="202.5" width="180" height="45"/>
+ <rect x="0" y="180" width="180" height="45" class="st8"/>
+ <text x="4.6" y="198.9" class="st14" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Hashing for lookup/Insertion into a <tspan
+ x="23.06" dy="1.2em" class="st15">vector of BFs happens once</tspan></text> </g>
+ <g id="shape44-157" v:mID="44" v:groupContext="shape" transform="translate(249.698,-151.505) rotate(-3.74012)">
+ <title>Sheet.44</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M-0 225 A93.4958 45.6256 42.23 0 1 79.38 221.66 L79.68 221.85" class="st16"/>
+ </g>
+ <g id="shape45-163" v:mID="45" v:groupContext="shape" transform="translate(30.375,0.25)">
+ <title>Sheet.45</title>
+ <desc>Lookup/Insertion is done in the series of BFs, one by one or ...</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="233.048" cy="202.5" width="466.1" height="45"/>
+ <rect x="0" y="180" width="466.096" height="45" class="st8"/>
+ <text x="4.34" y="206.1" class="st14" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Lookup/Insertion is done in the series of BFs, one by one or can be optimized to do in parallel. </text> </g>
+ <g id="shape46-166" v:mID="46" v:groupContext="shape" transform="translate(123.252,-43.6868) rotate(17.0249)">
+ <title>Sheet.46</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M-0 225 A88.2185 43.0621 47.63 0 1 70.31 221.39 L70.6 221.6" class="st16"/>
+ </g>
+ </g>
+ </g>
+</svg>
diff --git a/doc/guides/prog_guide/img/member_i5.svg b/doc/guides/prog_guide/img/member_i5.svg
new file mode 100644
index 0000000..c1728cf
--- /dev/null
+++ b/doc/guides/prog_guide/img/member_i5.svg
@@ -0,0 +1,163 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.0//EN" "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd">
+<!-- Generated by Microsoft Visio 11.0, SVG Export, v1.0 memship_i5.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+ xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="5.30481in" height="1.96146in"
+ viewBox="0 0 381.946 141.225" xml:space="preserve" color-interpolation-filters="sRGB" class="st15">
+ <v:documentProperties v:langID="1033" v:viewMarkup="false"/>
+
+ <style type="text/css">
+ <![CDATA[
+ .st1 {visibility:visible}
+ .st2 {fill:#5b9bd5;fill-opacity:0.22;stroke:#5b9bd5;stroke-opacity:0.22;stroke-width:0.25}
+ .st3 {fill:#5b9bd5;stroke:#c8c8c8;stroke-width:0.25}
+ .st4 {fill:none;stroke:none;stroke-width:0.25}
+ .st5 {fill:#ffffff;font-family:Calibri;font-size:1.16666em;font-weight:bold}
+ .st6 {marker-end:url(#mrkr5-14);stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st7 {fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-opacity:1;stroke-width:0.28409090909091}
+ .st8 {fill:#5b9bd5;font-family:Calibri;font-size:0.833336em}
+ .st9 {fill:#5b9bd5;font-family:Calibri;font-size:0.75em}
+ .st10 {stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st11 {marker-end:url(#mrkr5-63);stroke:#000000;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.75}
+ .st12 {fill:#000000;fill-opacity:1;stroke:#000000;stroke-opacity:1;stroke-width:0.22935779816514}
+ .st13 {fill:#000000;font-family:Calibri;font-size:1.00001em}
+ .st14 {font-size:1em}
+ .st15 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+ ]]>
+ </style>
+
+ <defs id="Markers">
+ <g id="lend5">
+ <path d="M 2 1 L 0 0 L 1.98117 -0.993387 C 1.67173 -0.364515 1.67301 0.372641 1.98465 1.00043 " style="stroke:none"/>
+ </g>
+ <marker id="mrkr5-14" class="st7" v:arrowType="5" v:arrowSize="2" v:setback="6.16" refX="-6.16" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-3.52,-3.52) "/>
+ </marker>
+ <marker id="mrkr5-63" class="st12" v:arrowType="5" v:arrowSize="2" v:setback="7.15" refX="-7.15" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-4.36,-4.36) "/>
+ </marker>
+ </defs>
+ <g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+ <title>Page-1</title>
+ <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+ <g id="group1-1" transform="translate(191.995,-19.4751)" v:mID="1" v:groupContext="group">
+ <title>Sheet.1</title>
+ <g id="shape2-2" v:mID="2" v:groupContext="shape" transform="translate(146.944,42.2251) rotate(90)">
+ <title>Triangle</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow2-3" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,1.9728,-0.3456)" class="st1">
+ <path d="M99 141.23 L49.5 58.62 L0 141.23 L99 141.23 Z" class="st2"/>
+ </g>
+ <path d="M99 141.23 L49.5 58.62 L0 141.23 L99 141.23 Z" class="st3"/>
+ </g>
+ <g id="shape3-6" v:mID="3" v:groupContext="shape" transform="translate(0,-34.65)">
+ <title>Sheet.3</title>
+ <desc>vBF</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="38.1251" cy="126.375" width="76.26" height="29.7"/>
+ <rect x="0" y="111.525" width="76.2502" height="29.7" class="st4"/>
+ <text x="27.68" y="130.58" class="st5" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>vBF </text> </g>
+ </g>
+ <g id="shape4-9" v:mID="4" v:groupContext="shape" transform="translate(126.724,-100.475)">
+ <title>Sheet.4</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 141.23 L64.83 141.23" class="st6"/>
+ </g>
+ <g id="shape5-15" v:mID="5" v:groupContext="shape" transform="translate(103.5,-101.775)">
+ <title>Sheet.5</title>
+ <desc>Flow Key</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="39.9122" cy="135.863" width="79.83" height="10.7251"/>
+ <rect x="0" y="130.5" width="79.8244" height="10.7251" class="st4"/>
+ <text x="21.78" y="138.86" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow Key</text> </g>
+ <g id="shape6-18" v:mID="6" v:groupContext="shape" transform="translate(221.726,-56.2468) rotate(-24.5123)">
+ <title>Sheet.6</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 141.23 L65.42 141.23" class="st6"/>
+ </g>
+ <g id="shape7-23" v:mID="7" v:groupContext="shape" transform="translate(280.318,-68.9751)">
+ <title>Sheet.7</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 141.23 L64.83 141.23" class="st6"/>
+ </g>
+ <g id="shape8-28" v:mID="8" v:groupContext="shape" transform="translate(338.125,-56.6022) rotate(24.1625)">
+ <title>Sheet.8</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 141.23 L70.8 141.23" class="st6"/>
+ </g>
+ <g id="shape9-33" v:mID="9" v:groupContext="shape" transform="translate(197.714,217.975) rotate(180)">
+ <title>Sheet.9</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 141.23 L51.03 141.23" class="st6"/>
+ </g>
+ <g id="shape10-38" v:mID="10" v:groupContext="shape" transform="translate(18,-67.5)">
+ <title>Sheet.10</title>
+ <desc>New Flow =&#62; New Assignment</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="80.4201" cy="134.475" width="160.85" height="13.5"/>
+ <rect x="0" y="127.725" width="160.84" height="13.5" class="st4"/>
+ <text x="25.11" y="137.18" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>New Flow =&#62; New Assignment</text> </g>
+ <g id="shape11-41" v:mID="11" v:groupContext="shape" transform="translate(198.032,253.975) rotate(180)">
+ <title>Sheet.11</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 141.23 L51.03 141.23" class="st6"/>
+ </g>
+ <g id="shape12-46" v:mID="12" v:groupContext="shape" transform="translate(0,-31.5)">
+ <title>Sheet.12</title>
+ <desc>Old Flow =&#62; forward to specific thread</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="81" cy="136.725" width="162.01" height="9"/>
+ <rect x="0" y="132.225" width="162" height="9" class="st4"/>
+ <text x="11.04" y="139.43" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Old Flow =&#62; forward to specific thread</text> </g>
+ <g id="shape13-49" v:mID="13" v:groupContext="shape" transform="translate(494.552,22.75) rotate(90)">
+ <title>Sheet.13</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 134.49 C3.18 142.89 7.57 142.28 11.25 138.99 C15.79 134.93 19.26 126.78 27 134.49" class="st10"/>
+ </g>
+ <g id="shape14-52" v:mID="14" v:groupContext="shape" transform="translate(494.552,58.75) rotate(90)">
+ <title>Sheet.14</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 134.49 C3.18 142.89 7.57 142.28 11.25 138.99 C15.79 134.93 19.26 126.78 27 134.49" class="st10"/>
+ </g>
+ <g id="shape15-55" v:mID="15" v:groupContext="shape" transform="translate(494.552,94.75) rotate(90)">
+ <title>Sheet.15</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 134.49 C3.18 142.89 7.57 142.28 11.25 138.99 C15.79 134.93 19.26 126.78 27 134.49" class="st10"/>
+ </g>
+ <g id="shape17-58" v:mID="17" v:groupContext="shape" transform="translate(348.769,-25.0593) rotate(44.5185)">
+ <title>Sheet.17</title>
+ <path d="M-0 141.23 A35.1884 19.2595 167.75 0 1 42.43 138.27 L42.74 138.46" class="st11"/>
+ </g>
+ <g id="shape18-64" v:mID="18" v:groupContext="shape" transform="translate(222.188,-5.40005)">
+ <title>Sheet.18</title>
+ <desc>A BF corresponding to each worker thread</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="59.0625" cy="127.725" width="118.13" height="27"/>
+ <rect x="0" y="114.225" width="118.125" height="27" class="st4"/>
+ <text x="5.14" y="124.13" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>A BF corresponding to <tspan
+ x="11.19" dy="1.2em" class="st14">each worker thread</tspan></text> </g>
+ </g>
+</svg>
diff --git a/doc/guides/prog_guide/img/member_i6.svg b/doc/guides/prog_guide/img/member_i6.svg
new file mode 100644
index 0000000..265179f
--- /dev/null
+++ b/doc/guides/prog_guide/img/member_i6.svg
@@ -0,0 +1,332 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.0//EN" "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd">
+<!-- Generated by Microsoft Visio 11.0, SVG Export, v1.0 memship_i6.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+ xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="8in" height="3.625in" viewBox="0 0 576 261"
+ xml:space="preserve" color-interpolation-filters="sRGB" class="st16">
+ <v:documentProperties v:langID="1033" v:viewMarkup="false"/>
+
+ <style type="text/css">
+ <![CDATA[
+ .st1 {visibility:visible}
+ .st2 {fill:#5b9bd5;fill-opacity:0.22;stroke:#5b9bd5;stroke-opacity:0.22;stroke-width:0.25}
+ .st3 {fill:#5b9bd5;stroke:#c8c8c8;stroke-width:0.25}
+ .st4 {fill:#feffff;font-family:Calibri;font-size:0.666664em;font-weight:bold}
+ .st5 {font-size:1em}
+ .st6 {fill:#70ad47;fill-opacity:0.5;stroke:#00b050;stroke-width:1.5}
+ .st7 {fill:none;stroke:none;stroke-width:0.25}
+ .st8 {fill:#00b050;font-family:Calibri;font-size:1.00001em}
+ .st9 {fill:none;stroke:#ff0000;stroke-width:0.25}
+ .st10 {fill:#5b9bd5;font-family:Calibri;font-size:0.833336em}
+ .st11 {marker-end:url(#mrkr5-29);stroke:#ff0000;stroke-linecap:round;stroke-linejoin:round;stroke-width:2.25}
+ .st12 {fill:#ff0000;fill-opacity:1;stroke:#ff0000;stroke-opacity:1;stroke-width:0.47169811320755}
+ .st13 {fill:#5b9bd5;font-family:Calibri;font-size:0.75em}
+ .st14 {fill:#92d050;stroke:#c8c8c8;stroke-width:0.25}
+ .st15 {fill:#5b9bd5;font-family:Calibri;font-size:1.00001em;font-weight:bold}
+ .st16 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+ ]]>
+ </style>
+
+ <defs id="Markers">
+ <g id="lend5">
+ <path d="M 2 1 L 0 0 L 1.98117 -0.993387 C 1.67173 -0.364515 1.67301 0.372641 1.98465 1.00043 " style="stroke:none"/>
+ </g>
+ <marker id="mrkr5-29" class="st12" v:arrowType="5" v:arrowSize="2" v:setback="3.71" refX="-3.71" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-2.12,-2.12) "/>
+ </marker>
+ </defs>
+ <g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+ <title>Page-1</title>
+ <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+ <g id="group121-1" transform="translate(21.0294,-9.8478)" v:mID="121" v:groupContext="group">
+ <title>Sheet.121</title>
+ <g id="shape49-2" v:mID="49" v:groupContext="shape" transform="translate(396.989,-54.9268)">
+ <title>Rectangle.2</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow49-3" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="219.549" width="99.4817" height="41.4507" class="st2"/>
+ </g>
+ <rect x="0" y="219.549" width="99.4817" height="41.4507" class="st3"/>
+ </g>
+ <g id="shape50-6" v:mID="50" v:groupContext="shape" transform="translate(248.261,-12.1936)">
+ <title>Rectangle.4</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow50-7" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="178.099" width="99.4817" height="82.9014" class="st2"/>
+ </g>
+ <rect x="0" y="178.099" width="99.4817" height="82.9014" class="st3"/>
+ </g>
+ <g id="shape52-10" v:mID="52" v:groupContext="shape" transform="translate(6.07514E-013,-29.0155)">
+ <title>Rectangle.10</title>
+ <desc>Signatures for target 1</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="49.7409" cy="225.767" width="99.49" height="70.4662"/>
+ <g id="shadow52-11" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="190.534" width="99.4817" height="70.4662" class="st2"/>
+ </g>
+ <rect x="0" y="190.534" width="99.4817" height="70.4662" class="st3"/>
+ <text x="26.54" y="223.37" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Signatures for<v:newlineChar/><tspan
+ x="36.73" dy="1.2em" class="st5">target </tspan>1</text> </g>
+ <g id="shape53-16" v:mID="53" v:groupContext="shape" transform="translate(239.971,-20.4837)">
+ <title>Sheet.53</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <rect x="0" y="248.565" width="116.062" height="12.4352" class="st6"/>
+ </g>
+ <g id="shape54-18" v:mID="54" v:groupContext="shape" transform="translate(353.649,-19.9346)">
+ <title>Sheet.54</title>
+ <desc>Match 1</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="33.1606" cy="254.782" width="66.33" height="12.4352"/>
+ <rect x="0" y="248.565" width="66.3211" height="12.4352" class="st7"/>
+ <text x="13.06" y="258.38" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Match 1</text> </g>
+ <g id="shape55-21" v:mID="55" v:groupContext="shape" transform="translate(216.989,-210.652)">
+ <title>Sheet.55</title>
+ <desc>Packet Payload</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="49.7409" cy="252.71" width="99.49" height="16.5803"/>
+ <rect x="0" y="244.42" width="99.4817" height="16.5803" class="st9"/>
+ <text x="19.04" y="255.71" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Packet Payload</text> </g>
+ <g id="shape56-24" v:mID="56" v:groupContext="shape" transform="translate(526.665,52.2365) rotate(90)">
+ <title>Sheet.56</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 261 L16.52 261" class="st11"/>
+ </g>
+ <g id="shape96-30" v:mID="96" v:groupContext="shape" transform="translate(-3.0294,-95.7818)">
+ <title>Sheet.96</title>
+ <desc>Attack Signature Length 1</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="51.75" cy="248.565" width="103.5" height="24.8704"/>
+ <rect x="0" y="236.13" width="103.5" height="24.8704" class="st7"/>
+ <text x="4.79" y="251.26" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Attack Signature Length 1</text> </g>
+ <g id="group114-33" transform="translate(228.359,-134.152)" v:mID="114" v:groupContext="group">
+ <title>Sheet.114</title>
+ <g id="group106-34" transform="translate(0,-24.8704)" v:mID="106" v:groupContext="group">
+ <title>Sheet.106</title>
+ <g id="shape100-35" v:mID="100" v:groupContext="shape" transform="translate(3.65707E-013,-12.4352)">
+ <title>Rectangle.100</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow100-36" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape101-39" v:mID="101" v:groupContext="shape" transform="translate(24.8704,-12.4352)">
+ <title>Rectangle.101</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow101-40" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape102-43" v:mID="102" v:groupContext="shape" transform="translate(49.7409,-12.4352)">
+ <title>Rectangle.102</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow102-44" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape103-47" v:mID="103" v:groupContext="shape">
+ <title>Rectangle.103</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow103-48" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape104-51" v:mID="104" v:groupContext="shape" transform="translate(24.8704,1.13687E-013)">
+ <title>Rectangle.104</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow104-52" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape105-55" v:mID="105" v:groupContext="shape" transform="translate(49.7409,1.13687E-013)">
+ <title>Rectangle.105</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow105-56" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ </g>
+ <g id="group107-59" v:mID="107" v:groupContext="group">
+ <title>Sheet.107</title>
+ <g id="shape108-60" v:mID="108" v:groupContext="shape" transform="translate(3.65707E-013,-12.4352)">
+ <title>Rectangle.100</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow108-61" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape109-64" v:mID="109" v:groupContext="shape" transform="translate(24.8704,-12.4352)">
+ <title>Rectangle.101</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow109-65" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape110-68" v:mID="110" v:groupContext="shape" transform="translate(49.7409,-12.4352)">
+ <title>Rectangle.102</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow110-69" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape111-72" v:mID="111" v:groupContext="shape">
+ <title>Rectangle.103</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <g id="shadow111-73" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st14"/>
+ </g>
+ <g id="shape112-76" v:mID="112" v:groupContext="shape" transform="translate(24.8704,1.13687E-013)">
+ <title>Rectangle.104</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow112-77" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st3"/>
+ </g>
+ <g id="shape113-80" v:mID="113" v:groupContext="shape" transform="translate(49.7409,1.13687E-013)">
+ <title>Rectangle.105</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <g id="shadow113-81" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st2"/>
+ </g>
+ <rect x="0" y="248.565" width="24.8704" height="12.4352" class="st14"/>
+ </g>
+ </g>
+ </g>
+ <g id="shape89-84" v:mID="89" v:groupContext="shape" transform="translate(398.644,-116.927) rotate(24.4696)">
+ <title>Sheet.89</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 261 L143.75 261" class="st11"/>
+ </g>
+ <g id="shape115-89" v:mID="115" v:groupContext="shape" transform="translate(116.062,-1.19371E-012)">
+ <title>Rectangle.115</title>
+ <desc>Signatures for target 2</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="49.7409" cy="211.259" width="99.49" height="99.4817"/>
+ <g id="shadow115-90" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="161.518" width="99.4817" height="99.4817" class="st2"/>
+ </g>
+ <rect x="0" y="161.518" width="99.4817" height="99.4817" class="st3"/>
+ <text x="26.54" y="208.86" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Signatures for<v:newlineChar/><tspan
+ x="36.73" dy="1.2em" class="st5">target </tspan>2</text> </g>
+ <g id="shape116-95" v:mID="116" v:groupContext="shape" transform="translate(117.989,-95.7818)">
+ <title>Sheet.116</title>
+ <desc>Attack Signature Length 2</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="51.9909" cy="248.565" width="103.99" height="24.8704"/>
+ <rect x="0" y="236.13" width="103.982" height="24.8704" class="st7"/>
+ <text x="5.03" y="251.26" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Attack Signature Length 2</text> </g>
+ <g id="shape118-98" v:mID="118" v:groupContext="shape" transform="translate(392.971,-90.217)">
+ <title>Sheet.118</title>
+ <desc>Attack Signature Length L</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="248.565" width="108" height="24.8704"/>
+ <rect x="0" y="236.13" width="108" height="24.8704" class="st7"/>
+ <text x="7.43" y="251.26" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Attack Signature Length L</text> </g>
+ <g id="shape119-101" v:mID="119" v:groupContext="shape" transform="translate(384.909,-64.9346)">
+ <title>Sheet.119</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <rect x="0" y="248.565" width="116.062" height="12.4352" class="st6"/>
+ </g>
+ <g id="shape120-103" v:mID="120" v:groupContext="shape" transform="translate(491.971,-64.9346)">
+ <title>Sheet.120</title>
+ <desc>Match 2</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="33.1606" cy="254.782" width="66.33" height="12.4352"/>
+ <rect x="0" y="248.565" width="66.3211" height="12.4352" class="st7"/>
+ <text x="13.06" y="258.38" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Match 2</text> </g>
+ <g id="shape85-106" v:mID="85" v:groupContext="shape" transform="translate(478.538,12.9307) rotate(65.6291)">
+ <title>Sheet.85</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 261 L109.61 261" class="st11"/>
+ </g>
+ <g id="shape117-111" v:mID="117" v:groupContext="shape" transform="translate(247.054,-91.2818)">
+ <title>Sheet.117</title>
+ <desc>Attack Signature Length X</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="52.7082" cy="248.565" width="105.42" height="24.8704"/>
+ <rect x="0" y="236.13" width="105.416" height="24.8704" class="st7"/>
+ <text x="5.7" y="251.26" class="st13" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Attack Signature Length X</text> </g>
+ </g>
+ <g id="shape122-114" v:mID="122" v:groupContext="shape" transform="translate(315.114,-164.13)">
+ <title>Sheet.122</title>
+ <desc>HTSS</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="26.943" cy="248.565" width="53.89" height="24.8704"/>
+ <rect x="0" y="236.13" width="53.8859" height="24.8704" class="st7"/>
+ <text x="14.52" y="252.16" class="st15" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>HTSS</text> </g>
+ </g>
+</svg>
diff --git a/doc/guides/prog_guide/img/member_i7.svg b/doc/guides/prog_guide/img/member_i7.svg
new file mode 100644
index 0000000..e23ae26
--- /dev/null
+++ b/doc/guides/prog_guide/img/member_i7.svg
@@ -0,0 +1,399 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.0//EN" "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd">
+<!-- Generated by Microsoft Visio 11.0, SVG Export, v1.0 memship_i7.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+ xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="8.5in" height="4.5in" viewBox="0 0 612 324"
+ xml:space="preserve" color-interpolation-filters="sRGB" class="st23">
+ <v:documentProperties v:langID="1033" v:viewMarkup="false"/>
+
+ <style type="text/css">
+ <![CDATA[
+ .st1 {visibility:visible}
+ .st2 {fill:#5b9bd5;fill-opacity:0.22;stroke:#5b9bd5;stroke-opacity:0.22;stroke-width:0.25}
+ .st3 {fill:#5b9bd5;stroke:#c8c8c8;stroke-width:0.25}
+ .st4 {fill:#feffff;font-family:Calibri;font-size:0.833336em;font-weight:bold}
+ .st5 {font-size:1em}
+ .st6 {fill:#70ad47;fill-opacity:0.5;stroke:#00b050;stroke-width:1.5}
+ .st7 {fill:none;stroke:none;stroke-width:0.25}
+ .st8 {fill:#00b050;font-family:Calibri;font-size:1.16666em}
+ .st9 {fill:none;stroke:#00b050;stroke-width:2.25}
+ .st10 {fill:#5b9bd5;font-family:Calibri;font-size:0.833336em}
+ .st11 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em}
+ .st12 {fill:#a8d08d;stroke:#c8c8c8;stroke-width:0.25}
+ .st13 {marker-end:url(#mrkr5-83);stroke:#ff0000;stroke-linecap:round;stroke-linejoin:round;stroke-width:2.25}
+ .st14 {fill:#ff0000;fill-opacity:1;stroke:#ff0000;stroke-opacity:1;stroke-width:0.47169811320755}
+ .st15 {marker-end:url(#mrkr5-95);stroke:#92d050;stroke-linecap:round;stroke-linejoin:round;stroke-width:2.25}
+ .st16 {fill:#92d050;fill-opacity:1;stroke:#92d050;stroke-opacity:1;stroke-width:0.47169811320755}
+ .st17 {fill:#00b050;font-family:Calibri;font-size:1.00001em;font-weight:bold}
+ .st18 {fill:#5b9bd5;font-family:Calibri;font-size:1.00001em}
+ .st19 {fill:none;stroke:#ff0000;stroke-width:2.25}
+ .st20 {fill:#ff0000;font-family:Calibri;font-size:1.00001em;font-weight:bold}
+ .st21 {marker-end:url(#mrkr5-123);stroke:#ff0000;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st22 {fill:#ff0000;fill-opacity:1;stroke:#ff0000;stroke-opacity:1;stroke-width:0.28409090909091}
+ .st23 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+ ]]>
+ </style>
+
+ <defs id="Markers">
+ <g id="lend5">
+ <path d="M 2 1 L 0 0 L 1.98117 -0.993387 C 1.67173 -0.364515 1.67301 0.372641 1.98465 1.00043 " style="stroke:none"/>
+ </g>
+ <marker id="mrkr5-83" class="st14" v:arrowType="5" v:arrowSize="2" v:setback="3.71" refX="-3.71" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-2.12,-2.12) "/>
+ </marker>
+ <marker id="mrkr5-95" class="st16" v:arrowType="5" v:arrowSize="2" v:setback="3.71" refX="-3.71" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-2.12,-2.12) "/>
+ </marker>
+ <marker id="mrkr5-123" class="st22" v:arrowType="5" v:arrowSize="2" v:setback="5.8" refX="-5.8" orient="auto"
+ markerUnits="strokeWidth" overflow="visible">
+ <use xlink:href="#lend5" transform="scale(-3.52,-3.52) "/>
+ </marker>
+ </defs>
+ <g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+ <title>Page-1</title>
+ <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+ <g id="group121-1" transform="translate(21.0294,-32.2733)" v:mID="121" v:groupContext="group">
+ <title>Sheet.121</title>
+ <g id="shape49-2" v:mID="49" v:groupContext="shape" transform="translate(460.471,-62.2267)">
+ <title>Rectangle.2</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow49-3" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="279" width="108" height="45" class="st2"/>
+ </g>
+ <rect x="0" y="279" width="108" height="45" class="st3"/>
+ </g>
+ <g id="shape50-6" v:mID="50" v:groupContext="shape" transform="translate(320.452,-18.123)">
+ <title>Rectangle.4</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow50-7" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="234" width="108" height="90" class="st2"/>
+ </g>
+ <rect x="0" y="234" width="108" height="90" class="st3"/>
+ </g>
+ <g id="shape52-10" v:mID="52" v:groupContext="shape" transform="translate(0,-31.5)">
+ <title>Rectangle.10</title>
+ <desc>Flow Keys Matching Mask 1</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="285.75" width="108" height="76.5"/>
+ <g id="shadow52-11" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="247.5" width="108" height="76.5" class="st2"/>
+ </g>
+ <rect x="0" y="247.5" width="108" height="76.5" class="st3"/>
+ <text x="12.56" y="282.75" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow Keys Matching <tspan
+ x="39.1" dy="1.2em" class="st5">Mask </tspan>1</text> </g>
+ <g id="shape53-16" v:mID="53" v:groupContext="shape" transform="translate(311.452,-27.123)">
+ <title>Sheet.53</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <rect x="0" y="310.5" width="126" height="13.5" class="st6"/>
+ </g>
+ <g id="shape54-18" v:mID="54" v:groupContext="shape" transform="translate(424.471,-26.2267)">
+ <title>Sheet.54</title>
+ <desc>Match</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="36" cy="317.25" width="72" height="13.5"/>
+ <rect x="0" y="310.5" width="72" height="13.5" class="st7"/>
+ <text x="17.68" y="321.45" class="st8" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Match</text> </g>
+ <g id="shape55-21" v:mID="55" v:groupContext="shape" transform="translate(261,-247.163)">
+ <title>Sheet.55</title>
+ <desc>Flow ID1</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="27.1728" cy="315" width="54.35" height="18"/>
+ <rect x="0" y="306" width="54.3456" height="18" class="st9"/>
+ <text x="9.52" y="318" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow ID1</text> </g>
+ <g id="shape96-24" v:mID="96" v:groupContext="shape" transform="translate(0,-109.783)">
+ <title>Sheet.96</title>
+ <desc>Flow Mask 1</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="319.5" width="108" height="9"/>
+ <rect x="0" y="315" width="108" height="9" class="st7"/>
+ <text x="18.51" y="323.7" class="st11" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow Mask 1</text> </g>
+ <g id="group114-27" transform="translate(247.5,-163.783)" v:mID="114" v:groupContext="group">
+ <title>Sheet.114</title>
+ <g id="group106-28" transform="translate(0,-27)" v:mID="106" v:groupContext="group">
+ <title>Sheet.106</title>
+ <g id="shape100-29" v:mID="100" v:groupContext="shape" transform="translate(0,-13.5)">
+ <title>Rectangle.100</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow100-30" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape101-33" v:mID="101" v:groupContext="shape" transform="translate(27,-13.5)">
+ <title>Rectangle.101</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow101-34" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape102-37" v:mID="102" v:groupContext="shape" transform="translate(54,-13.5)">
+ <title>Rectangle.102</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow102-38" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape103-41" v:mID="103" v:groupContext="shape">
+ <title>Rectangle.103</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow103-42" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape104-45" v:mID="104" v:groupContext="shape" transform="translate(27,0)">
+ <title>Rectangle.104</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow104-46" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape105-49" v:mID="105" v:groupContext="shape" transform="translate(54,0)">
+ <title>Rectangle.105</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow105-50" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ </g>
+ <g id="group107-53" v:mID="107" v:groupContext="group">
+ <title>Sheet.107</title>
+ <g id="shape108-54" v:mID="108" v:groupContext="shape" transform="translate(0,-13.5)">
+ <title>Rectangle.100</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow108-55" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape109-58" v:mID="109" v:groupContext="shape" transform="translate(27,-13.5)">
+ <title>Rectangle.101</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow109-59" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape110-62" v:mID="110" v:groupContext="shape" transform="translate(54,-13.5)">
+ <title>Rectangle.102</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <g id="shadow110-63" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st12"/>
+ </g>
+ <g id="shape111-66" v:mID="111" v:groupContext="shape">
+ <title>Rectangle.103</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <g id="shadow111-67" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape112-70" v:mID="112" v:groupContext="shape" transform="translate(27,0)">
+ <title>Rectangle.104</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <g id="shadow112-71" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ <g id="shape113-74" v:mID="113" v:groupContext="shape" transform="translate(54,0)">
+ <title>Rectangle.105</title>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <g id="shadow113-75" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728"
+ v:shadowType="1" transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="310.5" width="27" height="13.5" class="st2"/>
+ </g>
+ <rect x="0" y="310.5" width="27" height="13.5" class="st3"/>
+ </g>
+ </g>
+ </g>
+ <g id="shape89-78" v:mID="89" v:groupContext="shape" transform="translate(413.723,393.802) rotate(146.31)">
+ <title>Sheet.89</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 324 L153.9 324" class="st13"/>
+ </g>
+ <g id="shape115-84" v:mID="115" v:groupContext="shape" transform="translate(126,0)">
+ <title>Rectangle.115</title>
+ <desc>Flow Keys Matching Mask 2</desc>
+ <v:userDefs>
+ <v:ud v:nameU="visVersion" v:val="VT0(15):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="270" width="108" height="108"/>
+ <g id="shadow115-85" v:groupContext="shadow" v:shadowOffsetX="0.3456" v:shadowOffsetY="-1.9728" v:shadowType="1"
+ transform="matrix(1,0,0,1,0.3456,1.9728)" class="st1">
+ <rect x="0" y="216" width="108" height="108" class="st2"/>
+ </g>
+ <rect x="0" y="216" width="108" height="108" class="st3"/>
+ <text x="12.56" y="267" class="st4" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow Keys Matching <tspan
+ x="39.1" dy="1.2em" class="st5">Mask </tspan>2</text> </g>
+ <g id="shape85-90" v:mID="85" v:groupContext="shape" transform="translate(635.321,91.2793) rotate(81.3573)">
+ <title>Sheet.85</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 324 L143.93 324" class="st15"/>
+ </g>
+ <g id="shape56-96" v:mID="56" v:groupContext="shape" transform="translate(579.175,-64.556) rotate(64.1257)">
+ <title>Sheet.56</title>
+ <path d="M0 324 L54.31 324" class="st15"/>
+ </g>
+ </g>
+ <g id="shape122-101" v:mID="122" v:groupContext="shape" transform="translate(351,-213.444)">
+ <title>Sheet.122</title>
+ <desc>HTSS with False Negative (Cache)</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="45" cy="304.722" width="90" height="38.556"/>
+ <rect x="0" y="285.444" width="90" height="38.556" class="st7"/>
+ <text x="13.29" y="301.72" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>HTSS with False <tspan
+ x="10.52" dy="1.2em" class="st5">Negative </tspan>(Cache)</text> </g>
+ <g id="shape123-105" v:mID="123" v:groupContext="shape" transform="translate(287.654,-290.556)">
+ <title>Sheet.123</title>
+ <desc>Active</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="24.1875" cy="310.5" width="48.38" height="27"/>
+ <rect x="0" y="297" width="48.375" height="27" class="st7"/>
+ <text x="8.63" y="314.1" class="st17" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Active</text> </g>
+ <g id="shape124-108" v:mID="124" v:groupContext="shape" transform="translate(278.827,-153)">
+ <title>Sheet.124</title>
+ <desc>Target for Flow ID 1</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="36.0864" cy="310.5" width="72.18" height="27"/>
+ <rect x="0" y="297" width="72.1728" height="27" class="st9"/>
+ <text x="11.93" y="306.9" class="st18" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Target for <tspan
+ x="13.54" dy="1.2em" class="st5">Flow ID </tspan>1</text> </g>
+ <g id="shape125-112" v:mID="125" v:groupContext="shape" transform="translate(155.857,-254.556)">
+ <title>Sheet.125</title>
+ <desc>Flow ID2</desc>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="27.1728" cy="315" width="54.35" height="18"/>
+ <rect x="0" y="306" width="54.3456" height="18" class="st19"/>
+ <text x="9.52" y="318" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow ID2</text> </g>
+ <g id="shape126-115" v:mID="126" v:groupContext="shape" transform="translate(153,-270)">
+ <title>Sheet.126</title>
+ <desc>New/Inactive</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="40.5" cy="310.5" width="81" height="27"/>
+ <rect x="0" y="297" width="81" height="27" class="st7"/>
+ <text x="6.77" y="314.1" class="st20" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>New/Inactive</text> </g>
+ <g id="shape127-118" v:mID="127" v:groupContext="shape" transform="translate(251.739,-239.709) rotate(14.0795)">
+ <title>Sheet.127</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 318.73 A39.2404 18 -180 0 0 49.73 320.91 L50.07 320.78" class="st21"/>
+ </g>
+ <g id="shape128-124" v:mID="128" v:groupContext="shape" transform="translate(219.24,-229.5)">
+ <title>Sheet.128</title>
+ <desc>Miss</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="20.88" cy="310.5" width="41.76" height="27"/>
+ <rect x="0" y="297" width="41.76" height="27" class="st7"/>
+ <text x="7.81" y="314.7" class="st11" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Miss</text> </g>
+ <g id="shape129-127" v:mID="129" v:groupContext="shape" transform="translate(147.029,-142.056)">
+ <title>Sheet.129</title>
+ <desc>Flow Mask 2</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="319.5" width="108" height="9"/>
+ <rect x="0" y="315" width="108" height="9" class="st7"/>
+ <text x="18.51" y="323.7" class="st11" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow Mask 2</text> </g>
+ <g id="shape130-130" v:mID="130" v:groupContext="shape" transform="translate(166.845,-18.5004) rotate(18.2325)">
+ <title>Sheet.130</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 293.46 A71.1913 104.269 -180 0 0 97.04 298.43 L97.25 298.14" class="st21"/>
+ </g>
+ <g id="shape131-135" v:mID="131" v:groupContext="shape" transform="translate(184.406,-3.04505) rotate(-3.24734)">
+ <title>Sheet.131</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 293.46 A112.345 104.269 -180 0 0 154.25 297.52 L154.52 297.28" class="st21"/>
+ </g>
+ <g id="shape132-140" v:mID="132" v:groupContext="shape" transform="translate(301.368,16.888) rotate(-25.868)">
+ <title>Sheet.132</title>
+ <v:userDefs>
+ <v:ud v:nameU="msvThemeColors" v:val="VT0(254):26"/>
+ </v:userDefs>
+ <path d="M0 293.46 A83.375 104.269 -180 0 0 113.91 298.14 L114.14 297.87" class="st21"/>
+ </g>
+ <g id="shape133-145" v:mID="133" v:groupContext="shape" transform="translate(345.029,-142.056)">
+ <title>Sheet.133</title>
+ <desc>Flow Mask X</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="319.5" width="108" height="9"/>
+ <rect x="0" y="315" width="108" height="9" class="st7"/>
+ <text x="18.43" y="323.7" class="st11" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow Mask X</text> </g>
+ <g id="shape134-148" v:mID="134" v:groupContext="shape" transform="translate(481.5,-139.5)">
+ <title>Sheet.134</title>
+ <desc>Flow Mask L</desc>
+ <v:textBlock v:margins="rect(4,4,4,4)"/>
+ <v:textRect cx="54" cy="319.5" width="108" height="9"/>
+ <rect x="0" y="315" width="108" height="9" class="st7"/>
+ <text x="19.12" y="323.7" class="st11" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Flow Mask L</text> </g>
+ </g>
+</svg>
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index 40f04a1..7ff5144 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -50,6 +50,7 @@ Programmer's Guide
timer_lib
hash_lib
efd_lib
+ member_lib
lpm_lib
lpm6_lib
packet_distrib_lib
@@ -191,6 +192,19 @@ Programmer's Guide

:numref:`figure_efd11` :ref:`figure_efd11`

+:numref:`figure_membership1` :ref:`figure_membership1`
+
+:numref:`figure_membership2` :ref:`figure_membership2`
+
+:numref:`figure_membership3` :ref:`figure_membership3`
+
+:numref:`figure_membership4` :ref:`figure_membership4`
+
+:numref:`figure_membership5` :ref:`figure_membership5`
+
+:numref:`figure_membership6` :ref:`figure_membership6`
+
+:numref:`figure_membership7` :ref:`figure_membership7`

**Tables**

diff --git a/doc/guides/prog_guide/member_lib.rst b/doc/guides/prog_guide/member_lib.rst
new file mode 100644
index 0000000..d8d4612
--- /dev/null
+++ b/doc/guides/prog_guide/member_lib.rst
@@ -0,0 +1,440 @@
+.. BSD LICENSE
+ Copyright(c) 2017 Intel Corporation. All rights reserved.
+ All rights reserved.
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions
+ are met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in
+ the documentation and/or other materials provided with the
+ distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ contributors may be used to endorse or promote products derived
+ from this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+.. _Member_Library:
+
+Membership Library
+=======================
+
+Introduction
+------------
+The DPDK Membership Library provides an API for DPDK applications to insert a
+new member, delete an existing member, or query the existence of a member in a
+given set, or a group of sets. For the case of a group of sets the library
+will return not only whether the element has been inserted before in one of
+the sets but also which set it belongs to. The Membership Library is an
+extension and generalization of a traditional filter structure (for example
+Bloom Filter [Member-bloom]) that has multiple usages in a wide variety of
+workloads and applications. In general, the Membership Library is a data
+structure that provides a “set-summary” on whether a member belongs to a set,
+and as discussed in details later, there are two advantages of using such a
+set-summary rather than operating on a “full-blown” complete list of elements:
+first, it has a much smaller storage requirement than storing the whole list of
+elements themselves, and secondly checking an element membership (or other
+operations) in this set-summary is much faster than checking it for the
+original full-blown complete list of elements.
+
+We use the term “Set-Summary” in this guide to refer to the space-efficient,
+probabilistic membership data structure that is provided by the library. A
+membership test for an element will return the set this element belongs to or
+the element is "not-found" with very high probability of accuracy. Set-summary
+is a fundamental data aggregation component that can be used in many network
+(and other) applications. It is a crucial structure to address performance and
+scalability issues of diverse network applications including overlay networks,
+data-centric networks, flow table summaries, network statistics and
+traffic monitoring. A set-summary is useful for applications who need to
+include a list of elements while a complete list requires too much space
+and/or too much processing cost. In these situations, the set-summary works as
+a lossy hash-based representation of a set of members. It can dramatically
+reduce space requirement and significantly improve the performance of set
+membership queries at the cost of introducing a very small membership test error
+probability.
+
+.. _figure_membership1:
+.. figure:: img/member_i1.*
+
+ Example Usages of Membership Library
+
+We believe that there are various usages for a Membership Library in a very
+large set of applications and workloads. Interested readers can refer to
+[Member-survey] for a survey of possible networking usages. The following figure
+provide a small set of examples of using the Membership Library. Sub-figure(a)
+depicts a distributed web cache architecture where a collection of proxies
+attempt to share their web caches (cached from a set of back-end web servers) to
+provide faster responses to clients, and the proxies uses the Membership
+Library to share summaries of what web pages/objects they are caching. With the
+Membership Library, a proxy receiving an \http request will inquire the
+set-summary to find its location and quickly determine whether to retrieve the
+requested web page from a nearby proxy or from a backend web server.
+Sub-figure(b) depicts another example for using the Membership Library to
+prevent routing loops which is typically done using slow TTL countdown and
+dropping packets when TTL expires. As shown in Sub-figure(b), an embedded
+set-summary in the packet header itself can be used to summarize the set of
+nodes a packet has gone through, and each node upon receiving a packet can check
+whether its id is a member of the set of visited nodes, and if it is then a
+routing loop is detected. Sub-Figure(c) presents another usage of Membership
+Library to load balance flows to worker threads with in-order guarantee where a
+set-summary is used to query if a packet belongs to an existing flow or a new
+flow. Packets belonging to a new flow are forwarded to the current least loaded
+worker thread, while those belonging to an existing flow are forwarded to the
+pre-assigned thread to guarantee in-order processing. Sub-figure(d) highlights
+yet another usage example in the database domain where a set-summary is used to
+determine joins between sets instead of creating a join by comparing each
+element of a set against the other elements in a different set, a join is done
+on the summaries since they can efficiently encode members of a given set.
+
+We are including a configurable Membership Library in DPDK to cover set
+membership functionality for both a single set and multi-set scenarios. The
+library is optimized to support the customer network applications which require
+membership checking functionality. In this guide, we will cover two set-summary
+schemes including vector of Bloom Filters and Hash-Table based
+set-summary schemes with and without false negative probability, followed by
+a brief discussion of the Membership Library API.
+
+Vector of Bloom Filters
+--------------------------
+
+Bloom Filter (BF) [Member-bloom] is a well-known space-efficient
+probabilistic data structure that answers set membership queries (test whether
+an element is a member of a set) with some probability of false positives and
+zero false negatives; a query for an element returns either it is "possibly in
+a set" (with very high probability) or "definitely not in a set".
+
+The BF is a method for representing a set of n elements (for example flow keys
+in network applications domain) to support membership queries. The idea of BF is
+to allocate a bit-vector v with m bits, which are initially all set to 0. Then
+it chooses k independent hash functions h1, h2, … hk with hash values range from
+1 to m to perform hashing calculations on each element. Every time when an
+element X being inserted into the set, the bits at positions h1(X), h2(X), …
+hk(X) in v are set to 1 (any particular bit might be set to 1 multiple times
+for multiple different inserted elements). Given a query for any element Y, the
+bits at positions h1(Y), h2(Y), ... hk(Y) are checked. If any of them is 0,
+then Y is definitely not in the set. Otherwise there is a high probability that
+Y is a member of the set with certain false positive probability. As shown in
+the next equation, the false positive probability can be made arbitrarily small
+by changing the number of hash functions (k) and the vector length (m).
+
+.. _figure_membership2:
+.. figure:: img/member_i2.*
+
+ Bloom Filter False Positive Probability
+
+Without BF, an accurate membership testing could involve a costly hash table
+lookup and full element comparison. The advantage of using a BF is to simplify
+the membership test into a series of hash calculations and memory accesses for a
+small bit-vector, which can be easily optimized. Hence the lookup throughput
+(set membership test) can be significantly faster than a normal hash table
+lookup with element comparison.
+
+.. _figure_membership3:
+.. figure:: img/member_i3.*
+
+ Detecting Routing Loops Using BF
+
+BF is used for applications that need only one set, and the
+membership of elements is checked against the BF. The example discussed
+in the above figure is one example of potential applications that uses only one
+set to capture the node IDs that have been visited so far by the packet. Each
+node will then check this embedded BF in the packet header for its own id, and
+if the BF indicates that the current node is definitely not in the set then a
+loop-free route is guaranteed.
+
+
+.. _figure_membership4:
+.. figure:: img/member_i4.*
+
+ Vector Bloom Filter (vBF) Overview
+
+To support membership test for both multiple sets and single set,
+the library implements Vector Bloom Filter (vBF) scheme.
+vBF basically composes multiple bloom filters as a vector of bloom filers.
+The membership test is conducted on all of the
+bloom filters concurrently to determine which set(s) it belongs to or none of
+them. The basic idea of vBF is shown in the above figure where an element is
+used to address multiple bloom filters concurrently and the bloom filter
+index(es) with a hit is returned.
+
+.. _figure_membership5:
+.. figure:: img/member_i5.*
+
+ vBF for Flow Scheduling to Worker Thread
+
+As previously mentioned, there are many usages of such structure. vBF is used
+for applications that needs to check membership against multiple sets
+simultaneously. The example discussed in the above figure uses a set to capture
+all flows being assigned for processing at a given worker thread. Upon receiving
+a packet the vBF is used to quickly figure out if this packet belongs to a new flow
+so as to be forwarded to the current least loaded worker thread, or otherwise it
+should be queued for an existing thread to guarantee in-order processing (i.e.
+the property of vBF to indicate right away that a given flow is a new one or
+not is critical to minimize response time latency).
+
+It should be noted that vBF can be implemented using a set of single bloom
+filters with sequential lookup of each BF. However, being able to concurrently
+search all set-summaries is a big throughput advantage. In the library, certain
+parallelism is realized by the implementation of checking all bloom filters
+together.
+
+
+Hash-Table based Set-Summaries
+---------------------------------
+
+Hash-table based set-summary (HTSS) is another scheme in the membership library.
+Cuckoo filter [Member-cfilter] is an example of HTSS.
+HTSS supports multi-set membership testing like what
+vBF does. However, while vBF is more adequate for a small number of targets, HTSS is more suitable
+and can easily outperform vBF when the number of sets is
+large, since HTSS uses a single hash table for membership testing while vBF
+requires testing a series of Bloom Filters each corresponding to one set.
+As a result, generally speaking vBF is more adequate for the case of a small limited number of sets
+while HTSS should be used with a larger number of sets. For example, based on our results comparing vBF and HTSS
+for the case of 1M keys and 5% maximum allowable false positive rate; vBF outperforms HTSS in memory footprint and
+update rate and was better or on-bar in terms of throughput for up to 16 target sets. It should be noted however,
+that actual performance results might be different and is based on the workload, configuration and the platform
+used, etc.
+
+
+.. _figure_membership6:
+.. figure:: img/member_i6.*
+
+ Using HTSS for Attack Signature Matching
+
+As shown in the above figure, attack signature matching where each set
+represents a certain signature length (for correctness of this example, an
+attack signature should not be a subset of another one) in the payload is a good
+example for using HTSS with 0% false negative (i.e., when an element returns not
+found, it has a 100% certainty that it is not a member of any set). The packet
+inspection application benefits from knowing right away that the current payload
+does not match any attack signatures in the database to establish its
+legitimacy, otherwise a deep inspection of the packet is needed.
+
+HTSS employs a similar but simpler data structure to a traditional hash table,
+and the major difference is that HTSS stores only the signatures but not the
+full keys/elements which can significantly reduce the footprint of the table.
+Along with the signature, HTSS also stores a value to indicate the target set.
+When looking up for an element, the element is hashed and the HTSS is addressed
+to retrieve the signature stored. If the signature matches then the value is
+retrieved corresponding to the index of the target set which the element belongs
+to. Because signatures can collide, HTSS can still has false positive
+probability. Furthermore, if elements are allowed to be
+overwritten or evicted when the hash table becomes full, it will also have a
+false negative probability. We discuss this case in the next section.
+
+Set-Summaries with False Negative Probability
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As previously mentioned, traditional set-summaries (e.g. Bloom Filters ) do not
+have a false negative probability, i.e., it is 100% certain when an element
+returns “not to be present” for a given set. However, the Membership Library
+also supports a set-summary probabilistic data structure based on HTSS which
+allows for false negative probability.
+
+
+In HTSS, when the hash table becomes full, keys/elements will fail to be added
+into the table and the hash table has to be resized to accommodate for these new
+elements, which can be expensive. However, if we allow new elements to overwrite
+or evict existing elements (as a cache typically does), then the resulting
+set-summary will begin to have false negative probability. This is because the
+element that was evicted from the set-summary may still be present in the target
+set. For subsequent inquiries the set-summary will falsely report the element
+not being in the set, hence having a false negative probability.
+
+The major usage of HTSS with false negative is to use it as a cache for
+distributing elements to different target sets. By allowing HTSS to evict old
+elements, the set-summary can keep track of the most recent elements
+(i.e. active) as a cache typically does. Old inactive elements (infrequently
+used elements) will automatically and eventually get evicted from the
+set-summary. It worth noting that the set-summary still has false positive
+probability, which means the application either can tolerate certain false positive
+or it has fall-back path when false positive happens.
+
+.. _figure_membership7:
+.. figure:: img/member_i7.*
+
+ Using HTSS with False Negatives for Wild Card Classification
+
+HTSS with false negative (i.e. a cache) also has its wide set of applications.
+For example wild card flow classification (e.g. ACL rules) highlighted in the
+above figure is an example of such application. In that case each target set
+represents a sub-table with rules defined by a certain flow mask. The flow masks
+are non-overlapping, and for flows matching more than one rule only the highest
+priority one is inserted in the corresponding sub-table (interested readers can
+refer to the Open vSwitch (OvS) design of Mega Flow Cache (MFC) [Member-OvS]
+for further details). Typically the rules will have a large number of distinct
+unique masks and hence, a large number of target sets each corresponding to one
+mask. Because the active set of flows varies widely based on the network
+traffic, HTSS with false negative will act as a cache for <flowid, target ACL
+sub-table> pair for the current active set of flows. When a miss occurs (as
+shown in red in the above figure) the sub-tables will be searched sequentially
+one by one for a possible match, and when found the flow key and target
+sub-table will be inserted into the set-summary (i.e. cache insertion) so
+subsequent packets from the same flow don’t incur the overhead of the
+sequential search of sub-tables.
+
+Library API Overview
+--------------------
+The design goal of the Membership Library API is to be as generic as possible to
+support all the different types of set-summaries we discussed in previous
+sections and beyond. Fundamentally, the APIs need to include creation,
+insertion, deletion, and lookup.
+
+.. Set-summary Type Query
+.. ~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. *void rte\_ms\_type\_query(enum rte\_ms\_setsum\_type\*\* types)*
+
+.. This function intends to serve as a convenient tool to query which set-summary
+.. types are supported in the current version of DPDK. Since the implementations of
+.. the set-summaries can vary, initial versions of the Membership Library may only
+.. support a subset of the set-summaries we discussed previously. Programmers could
+.. use this API to decide which type is available to use in the current version of DPDK.
+
+Set-summary Create
+~~~~~~~~~~~~~~~~~~~~~
+
+*rte\_member\_create* function is used to create a set-summary structure, the input parameter
+is a struct to pass in parameters that needed to initialize the set-summary, while the function returns the
+pointer to the created set-summary or NULL if the creation failed.
+
+The general input arguments used when creating the set-summary should include *name*
+which is the name of the created set-summary, *type* which is one of the types
+supported by the library (e.g. RTE\_MEMBER\_TYPE\_HT for HTSS or RTE\_MEMBER\_TYPE\_VBF for vBF), and *key\_len*
+which is the length of the element/key. There are other parameters
+are only used for certain type of set-summary, or has a slightly different meaning for different types of set-summary.
+For example, *num\_keys* parameter means the maximum number of entries for Hash table based set-summary.
+However, for bloom filter, this value means the expected number of keys that could be
+inserted into the bloom filter(s). The value is used to calculate the size of each
+bloom filter.
+We also pass two seeds: *prim\_hash\_seed* and
+*sec\_hash\_seed* for the primary and secondary hash functions to calculate two independent hash values.
+*socket\_id* parameter is the NUMA socket ID for the memory used to create the
+set-summary. For HTSS, another parameter *iscache* is used to indicate
+if this set-summary is a cache (i.e. with false negative probability) or not.
+For vBF, extra parameters are needed. For example, *num\_set* is the number of
+sets needed to initialize the vector bloom filters. This number is equal to the
+number of bloom filters will be created.
+*false\_pos\_rate* is the false positive rate. num\_keys and false\_pos\_rate will be used to determine
+the number of hash functions and the bloom filter size.
+
+
+Set-summary Element Insertion
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. *rte\_membership\_add(const void *setsum, const void *key, MEMBERSHIP\_TARGET\_TYPE target\_id)*
+
+*rte\_member\_add* is the function to insert an element/key in a set-summary structure, if it fails an
+error is returned. For success the returned value is deferent based on the
+set-summary mode to provide extra information for the users. For vBF
+mode, a return value of 0 means a successful insert. For HTSS mode without false negative, the insert
+could fail with -ENOSPC if the table is full. With false negative (i.e. cache mode),
+for insert that does not cause any eviction (i.e. no overwriting happens to an
+existing entry) the return value is 0. For insertion that causes eviction, the return
+value is 1 to indicate such situation, but it is not an error.
+
+The input arguments for the function should include the *key* which is a pointer to the element/key that needs to
+be added to the set-summary, and *set\_id* which is the set id associated
+with the key that needs to be added.
+
+
+Set-summary Element Lookup
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+.. *rte\_membership\_lookup(const void \*setsum, const void \*key, MEMBERSHIP\_TARGET\_TYPE \*target\_id)*
+
+*rte\_member\_lookup* looks up a single key/element in the set-summary structure. It
+returns as soon as the first match is found. The return value is 1 if a
+match is found and 0 otherwise. The arguments for the function include *key* which is a pointer to the
+element/key that needs to be looked up, and *set\_id* which is used to return the
+first target set id where the key has matched, if any.
+
+.. *rte\_membership\_lookup\_bulk(const void \*setsum, const void \*\*keys, uint32\_t num\_keys, MEMBERSHIP\_TARGET\_TYPE \*target\_ids)*
+
+*rte\_member\_lookup\_bulk* is the function to look up a bulk of keys/elements in the
+set-summary structure for their first match. Each key lookup returns as soon as the first match is found. The
+return value is the number of keys that find a match. The arguments of the
+function include *keys* which is a pointer to a bulk of keys that are to be looked up,
+*num\_keys* is the number
+of keys that will be looked up, and *set\_ids* are the return target set
+ids for the first match found for each of the input keys. *set\_ids* is an array
+needs to be sized according to the *num\_keys*. If there is no match, the set id
+for that key will be set to RTE_MEMBER_NO_MATCH.
+
+.. *rte\_membership\_lookup\_multi(const void \*setsum, const void \*key, MEMBERSHIP\_TARGET\_TYPE \*target\_id)*
+
+*rte\_member\_lookup\_multi* function looks up a single key/element in the
+set-summary structure for multiple matches. It
+returns ALL the matches (possibly more than one) found for this key when it
+is matched against all target sets (it worth noting that for cache mode HTSS,
+the current implementation matches at most one target set). The return value is
+the number of matches
+that was found for this key (for cache mode HTSS the return value
+should be at most 1). The arguments for the function include *key* which is a pointer to the
+element/key that needs to be looked up, *match\_per\_key* which is to indicate maximum number of matches
+the user expect to find for each key, and *set\_id* which is used to return all
+target set ids where the key has matched, if any. The *set\_id* array should be sized
+according to *match\_per\_key*. For vBF, maximum number of matches per key is equal
+to the number of sets. For HTSS, maximum number of matches per key is equal to two times
+entry count per bucket. *match\_per\_key* should be equal or smaller than maximum number of
+possible matches.
+
+.. *rte\_membership\_lookup\_multi\_bulk(const void \*setsum, const void \*\*keys, uint32\_t num\_keys, uint32\_t max\_match\_per\_key, uint32\_t \*match\_count, MEMBERSHIP\_TARGET\_TYPE \*target\_ids)*
+
+*rte\_membership\_lookup\_multi\_bulk* function looks up a bulk of keys/elements elements in the
+set-summary structure for multiple matches, each key lookup returns ALL the matches (possibly more
+than one) found for this key when it is matched against all target sets (cache mode HTSS
+matches at most one target set). The
+return value is the number of keys that find one or more matches in the
+set-summary structure. The arguments of the
+function include *keys* which is
+a pointer to a bulk of keys that are to be looked up, *num\_keys* is the number
+of keys that will be looked up, *match\_per\_key* is the possible
+max number of matches for each key, *match\_count* which is the returned number
+of matches for each key, and *set\_ids* are the returned target set
+ids for all matches found for each keys. *set\_ids* is 2-D array
+that for each key, a 1-D array should be sized according to *match\_per\_key*.
+*match\_per\_key* should be equal or smaller than maximum number of
+possible matches, similar to *rte\_member\_lookup\_multi*.
+
+
+Set-summary Element Delete
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. *rte\_membership\_delete(void \*setsum, const void \*key, MEMBERSHIP\_TARGET\_TYPE target\_id)*
+
+*rte\_membership\_delete* function deletes an element/key from a set-summary structure, if it fails
+an error is returned. The input arguments should include *key* which is a pointer to the
+element/key that needs to be deleted from the set-summary, and *set\_id*
+which is the set id associated with the key to delete. It worth noting that current
+implementation of vBF does not support deletion [1]_. An error code -EINVAL will be returned.
+
+.. [1] Traditional bloom filter does not support proactive deletion. Supporting proactive deletion require additional implementation and performance overhead.
+
+References
+-----------
+
+[Member-bloom] B H Bloom, "Space/Time Trade-offs in Hash Coding with Allowable Errors," Communications of the ACM, 1970.
+
+[Member-survey] A Broder and M Mitzenmacher, "Network Applications of Bloom Filters: A Survey," in Internet Mathematics, 2005.
+
+[Member-cfilter] B Fan, D G Andersen and M Kaminsky, "Cuckoo Filter: Practically Better Than Bloom," in Conference on emerging Networking Experiments and Technologies, 2014.
+
+[Member-OvS] B Pfaff, "The Design and Implementation of Open vSwitch," in NSDI, 2015.
\ No newline at end of file
diff --git a/doc/guides/rel_notes/release_17_11.rst b/doc/guides/rel_notes/release_17_11.rst
index 170f4f9..4002df3 100644
--- a/doc/guides/rel_notes/release_17_11.rst
+++ b/doc/guides/rel_notes/release_17_11.rst
@@ -41,6 +41,23 @@ New Features
Also, make sure to start the actual text at the margin.
=========================================================

+* **Added Membership library (rte_member).**
+
+ Added membership library. It provides an API for DPDK applications to insert a
+ new member, delete an existing member, or query the existence of a member in a
+ given set, or a group of sets. For the case of a group of sets the library
+ will return not only whether the element has been inserted before in one of
+ the sets but also which set it belongs to.
+
+ The Membership Library is an extension and generalization of a traditional
+ filter (for example Bloom Filter) structure that has multiple usages in a wide
+ variety of workloads and applications. In general, the Membership Library is a
+ data structure that provides a “set-summary” and responds to set-membership
+ queries whether a certain member belongs to a set(s).
+
+ See the :ref:`Membership Library <Member_Library>` documentation in
+ the Programmers Guide document, for more information.
+

Resolved Issues
---------------
--
2.7.4
Mcnamara, John
2017-09-04 13:19:07 UTC
Permalink
-----Original Message-----
Sent: Saturday, September 2, 2017 2:25 AM
Subject: [dpdk-dev] [PATCH v2 7/7] doc: add membership documentation
This patch adds the documentation for membership library.
---
Thanks for the detailed docs and the nice images. Some comments below.
+
+
+Membership Library
+=======================
The underline should match the length of the title and should be followed by
a blank line. Make this change throughout the doc.
+
+Introduction
+------------
+The DPDK Membership Library provides an API for DPDK applications to insert a
+new member, delete an existing member, or query the existence of a member in a
+given set, or a group of sets. For the case of a group of sets the library
+will return not only whether the element has been inserted before in one of
+the sets but also which set it belongs to. The Membership Library is an
+extension and generalization of a traditional filter structure (for example
+Bloom Filter [Member-bloom]) that has multiple usages in a wide variety of
+workloads and applications. In general, the Membership Library is a data
+structure that provides a “set-summary” on whether a member belongs to a set,
Use standard quotes instead of smart quotes. The doc renderers will convert
them to smart quotes.
+probability.
+
+.. figure:: img/member_i1.*
+
+ Example Usages of Membership Library
+
+We believe that there are various usages for a Membership Library in a very
+large set of applications and workloads. Interested readers can refer to
+[Member-survey] for a survey of possible networking usages. The following figure
+provide a small set of examples of using the Membership Library. Sub-
figure(a)
These sub-figures would be better as separate bullet sections.

There are a lot of other small fixes that I will send on to you off-line.


Re
Yipeng Wang
2017-09-05 23:59:42 UTC
Permalink
DPDK Membership Library provides an API that can be used by many DPDK
applications to conduct one or more set-membership tests (we mention some
possible use cases below, but interested readers can refer to
[1] for a wider survey of use cases).

The basic functionalities of the Membership Library include
inserting a new member, deleting an existing member, and querying the existence
of a member. The query result would indicate with high accuracy which specific
set this member belongs to among a group of sets.

The Membership Library is an extension and generalization of traditional filter
data structures [2,3], which maintain a space-efficient “set-summary”.
There are two advantages of using such a set-summary rather than operating on a
“full-blown” complete list of elements: firstly it has a much smaller storage
requirement than storing the whole list of elements, and secondly set membership
tests (or other operations) is much more efficient than searching through the
complete list of elements.

A membership test for an element will return the set this element belongs to if
found (or return "not-found") with high accuracy. If needed, the accuracy of the
membership tests could be further increased with larger storage space.
Set-summary is a fundamental data aggregation component that can be used in many
network applications. It is a crucial structure to address performance and
scalability issues of diverse applications including overlay networks, wild card
flow classification, web-caches, load balancing, connection tracking,
data-centric networks, flow table summaries, network statistics and traffic
monitoring. Our Proof of Concept (PoC) using set-summary to optimize flow lookup
in Open vSwitch (OvS) shows a speedup of about 2-3X.

This patch set implements two types of set-summaries, i.e., hash-table based
set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the
non-cache and cache modes. The non-cache mode can incur a small chance of
false-positives which is the case when the set-summary indicates a key belongs
to a given set while actually it is not. The cache mode can also have
false-negatives in addition to false-positives. False-negatives means the case
when the set-summary indicates a key does not belong to a given set while
actually it does. This happens because cache mode allows new key to evict
existing keys. vBF only has false-positives similar to the non-cache HTSS.
However, one can set the false-positive rate arbitrarily. HTSS's
false-positive rate is determined by the hash-table size and the signature size.

[1] A Broder and M Mitzenmacher, “Network Applications of Bloom Filters: A
Survey,” in Internet Mathematics, 2005.

[2] B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,”
Communications of the ACM, 1970.

[3] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than
Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.


v3:
- documentation: update documentation to incorperate John's review.

v2:
- test: add bad parameter test functions to test the creation fail case.
- test: add find existing test.
- member lib: Add num_entries check in rte_member_create_ht.
- member lib: Add multiple checks for rte_member_create_vbf to avoid divide-by-0.
- member lib: remove unnecessary ret from rte_member.c according to Stephen's
comment.
- member lib: change the vBF creation function to be not too conservative.
Previous algorithm is too conservative on false positive.
- member lib: fix uninitilize issue fail gcc-4.8 in rte_member_find_existing.
- member lib: add rte_member_find_existing to version map.
- makefile: lib/librte_member/Makefile: change include order according to
Luca's comment.
- documentation: update the programmer guide for membership library.

Yipeng Wang (7):
member: implement main API
member: implement HT mode
member: implement vBF mode
member: add AVX for HT mode
member: enable the library
test/member: add functional and perf tests
doc: add membership documentation

MAINTAINERS | 7 +
config/common_base | 5 +
doc/api/doxy-api-index.md | 3 +-
doc/api/doxy-api.conf | 1 +
doc/guides/prog_guide/img/member_i1.svg | 1613 ++++++++++++++++++++++++++++++
doc/guides/prog_guide/img/member_i2.svg | 36 +
doc/guides/prog_guide/img/member_i3.svg | 148 +++
doc/guides/prog_guide/img/member_i4.svg | 450 +++++++++
doc/guides/prog_guide/img/member_i5.svg | 163 +++
doc/guides/prog_guide/img/member_i6.svg | 332 ++++++
doc/guides/prog_guide/img/member_i7.svg | 399 ++++++++
doc/guides/prog_guide/index.rst | 14 +
doc/guides/prog_guide/member_lib.rst | 421 ++++++++
doc/guides/rel_notes/release_17_11.rst | 17 +
lib/Makefile | 2 +
lib/librte_eal/common/eal_common_log.c | 1 +
lib/librte_eal/common/include/rte_log.h | 1 +
lib/librte_member/Makefile | 51 +
lib/librte_member/rte_member.c | 342 +++++++
lib/librte_member/rte_member.h | 518 ++++++++++
lib/librte_member/rte_member_ht.c | 569 +++++++++++
lib/librte_member/rte_member_ht.h | 115 +++
lib/librte_member/rte_member_vbf.c | 350 +++++++
lib/librte_member/rte_member_vbf.h | 85 ++
lib/librte_member/rte_member_version.map | 16 +
lib/librte_member/rte_member_x86.h | 111 ++
mk/rte.app.mk | 2 +
test/test/Makefile | 3 +
test/test/test_member.c | 682 +++++++++++++
test/test/test_member_perf.c | 643 ++++++++++++
30 files changed, 7099 insertions(+), 1 deletion(-)
create mode 100644 doc/guides/prog_guide/img/member_i1.svg
create mode 100644 doc/guides/prog_guide/img/member_i2.svg
create mode 100644 doc/guides/prog_guide/img/member_i3.svg
create mode 100644 doc/guides/prog_guide/img/member_i4.svg
create mode 100644 doc/guides/prog_guide/img/member_i5.svg
create mode 100644 doc/guides/prog_guide/img/member_i6.svg
create mode 100644 doc/guides/prog_guide/img/member_i7.svg
create mode 100644 doc/guides/prog_guide/member_lib.rst
create mode 100644 lib/librte_member/Makefile
create mode 100644 lib/librte_member/rte_member.c
create mode 100644 lib/librte_member/rte_member.h
create mode 100644 lib/librte_member/rte_member_ht.c
create mode 100644 lib/librte_member/rte_member_ht.h
create mode 100644 lib/librte_member/rte_member_vbf.c
create mode 100644 lib/librte_member/rte_member_vbf.h
create mode 100644 lib/librte_member/rte_member_version.map
create mode 100644 lib/librte_member/rte_member_x86.h
create mode 100644 test/test/test_member.c
create mode 100644 test/test/test_member_perf.c
--
2.7.4
Yipeng Wang
2017-09-05 23:59:43 UTC
Permalink
Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a
set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the
set-summary.

The results of the membership test is not 100% accurate. Certain
false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set-summary"
is memory efficient and fast on lookup.

This patch add the main API definition.

Signed-off-by: Yipeng Wang <***@intel.com>
---
lib/Makefile | 2 +
lib/librte_eal/common/eal_common_log.c | 1 +
lib/librte_eal/common/include/rte_log.h | 1 +
lib/librte_member/Makefile | 49 +++
lib/librte_member/rte_member.c | 342 ++++++++++++++++++++
lib/librte_member/rte_member.h | 518 +++++++++++++++++++++++++++++++
lib/librte_member/rte_member_version.map | 16 +
7 files changed, 929 insertions(+)
create mode 100644 lib/librte_member/Makefile
create mode 100644 lib/librte_member/rte_member.c
create mode 100644 lib/librte_member/rte_member.h
create mode 100644 lib/librte_member/rte_member_version.map

diff --git a/lib/Makefile b/lib/Makefile
index 86caba1..c82033a 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -108,6 +108,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += librte_reorder
DEPDIRS-librte_reorder := librte_eal librte_mempool librte_mbuf
DIRS-$(CONFIG_RTE_LIBRTE_PDUMP) += librte_pdump
DEPDIRS-librte_pdump := librte_eal librte_mempool librte_mbuf librte_ether
+DIRS-$(CONFIG_RTE_LIBRTE_MEMBER) += librte_member
+DEPDIRS-librte_member := librte_eal librte_hash

ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y)
DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
diff --git a/lib/librte_eal/common/eal_common_log.c b/lib/librte_eal/common/eal_common_log.c
index 0e3b932..90db6a3 100644
--- a/lib/librte_eal/common/eal_common_log.c
+++ b/lib/librte_eal/common/eal_common_log.c
@@ -279,6 +279,7 @@ static const struct logtype logtype_strings[] = {
{RTE_LOGTYPE_CRYPTODEV, "cryptodev"},
{RTE_LOGTYPE_EFD, "efd"},
{RTE_LOGTYPE_EVENTDEV, "eventdev"},
+ {RTE_LOGTYPE_MEMBER, "member"},
{RTE_LOGTYPE_USER1, "user1"},
{RTE_LOGTYPE_USER2, "user2"},
{RTE_LOGTYPE_USER3, "user3"},
diff --git a/lib/librte_eal/common/include/rte_log.h b/lib/librte_eal/common/include/rte_log.h
index ec8dba7..ce1a3d0 100644
--- a/lib/librte_eal/common/include/rte_log.h
+++ b/lib/librte_eal/common/include/rte_log.h
@@ -87,6 +87,7 @@ extern struct rte_logs rte_logs;
#define RTE_LOGTYPE_CRYPTODEV 17 /**< Log related to cryptodev. */
#define RTE_LOGTYPE_EFD 18 /**< Log related to EFD. */
#define RTE_LOGTYPE_EVENTDEV 19 /**< Log related to eventdev. */
+#define RTE_LOGTYPE_MEMBER 20 /**< Log related to membership. */

/* these log types can be used in an application */
#define RTE_LOGTYPE_USER1 24 /**< User-defined log type 1. */
diff --git a/lib/librte_member/Makefile b/lib/librte_member/Makefile
new file mode 100644
index 0000000..1a79eaa
--- /dev/null
+++ b/lib/librte_member/Makefile
@@ -0,0 +1,49 @@
+# BSD LICENSE
+#
+# Copyright(c) 2017 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in
+# the documentation and/or other materials provided with the
+# distribution.
+# * Neither the name of Intel Corporation nor the names of its
+# contributors may be used to endorse or promote products derived
+# from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS := -I$(SRCDIR) $(CFLAGS)
+CFLAGS += $(WERROR_FLAGS) -O3
+
+EXPORT_MAP := rte_member_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_member/rte_member.c b/lib/librte_member/rte_member.c
new file mode 100644
index 0000000..71b066d
--- /dev/null
+++ b/lib/librte_member/rte_member.c
@@ -0,0 +1,342 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <string.h>
+
+#include <rte_eal.h>
+#include <rte_eal_memconfig.h>
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_malloc.h>
+#include <rte_errno.h>
+
+#include "rte_member.h"
+#include "rte_member_ht.h"
+#include "rte_member_vbf.h"
+
+TAILQ_HEAD(rte_member_list, rte_tailq_entry);
+static struct rte_tailq_elem rte_member_tailq = {
+ .name = "RTE_MEMBER",
+};
+EAL_REGISTER_TAILQ(rte_member_tailq)
+
+
+void *
+rte_member_find_existing(const char *name)
+{
+ struct rte_member_setsum *setsum = NULL;
+ struct rte_tailq_entry *te;
+ struct rte_member_list *member_list;
+
+ member_list = RTE_TAILQ_CAST(rte_member_tailq.head, rte_member_list);
+
+ rte_rwlock_read_lock(RTE_EAL_TAILQ_RWLOCK);
+ TAILQ_FOREACH(te, member_list, next) {
+ setsum = (struct rte_member_setsum *) te->data;
+ if (strncmp(name, setsum->name, RTE_MEMBER_NAMESIZE) == 0)
+ break;
+ }
+ rte_rwlock_read_unlock(RTE_EAL_TAILQ_RWLOCK);
+
+ if (te == NULL) {
+ rte_errno = ENOENT;
+ return NULL;
+ }
+ return setsum;
+}
+
+void
+rte_member_free(void *ss)
+{
+ struct rte_member_setsum *setsum;
+ struct rte_member_list *member_list = NULL;
+ struct rte_tailq_entry *te;
+
+ if (ss == NULL)
+ return;
+ setsum = ss;
+ member_list = RTE_TAILQ_CAST(rte_member_tailq.head, rte_member_list);
+ rte_rwlock_write_lock(RTE_EAL_TAILQ_RWLOCK);
+ TAILQ_FOREACH(te, member_list, next) {
+ if (te->data == ss)
+ break;
+ }
+ if (te == NULL) {
+ rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+ return;
+ }
+ TAILQ_REMOVE(member_list, te, next);
+ rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ rte_member_free_ht(setsum);
+ break;
+ case RTE_MEMBER_TYPE_VBF:
+ rte_member_free_vbf(setsum);
+ break;
+ default:
+ break;
+ }
+ rte_free(setsum);
+ rte_free(te);
+}
+
+
+void *
+rte_member_create(const struct rte_member_parameters *params)
+{
+ struct rte_tailq_entry *te;
+ struct rte_member_list *member_list = NULL;
+ struct rte_member_setsum *setsum = NULL;
+ int ret;
+
+ if (params == NULL) {
+ rte_errno = EINVAL;
+ return NULL;
+ }
+
+ if ((params->key_len == 0)) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, MEMBER,
+ "Memship create with invalid parameters\n");
+ return NULL;
+ }
+
+ member_list = RTE_TAILQ_CAST(rte_member_tailq.head, rte_member_list);
+
+ rte_rwlock_write_lock(RTE_EAL_TAILQ_RWLOCK);
+
+ TAILQ_FOREACH(te, member_list, next) {
+ setsum = (struct rte_member_setsum *) te->data;
+ if (strncmp(params->name, setsum->name,
+ RTE_MEMBER_NAMESIZE) == 0)
+ break;
+ }
+ setsum = NULL;
+ if (te != NULL) {
+ rte_errno = EEXIST;
+ te = NULL;
+ goto error_unlock_exit;
+ }
+ te = rte_zmalloc("MEMBER_TAILQ_ENTRY", sizeof(*te), 0);
+ if (te == NULL) {
+ RTE_LOG(ERR, MEMBER, "tailq entry allocation failed\n");
+ goto error_unlock_exit;
+ }
+
+ /* Create a new setsum structure */
+ setsum = (struct rte_member_setsum *) rte_zmalloc_socket(params->name,
+ sizeof(struct rte_member_setsum), RTE_CACHE_LINE_SIZE,
+ params->socket_id);
+ if (setsum == NULL) {
+ RTE_LOG(ERR, MEMBER, "Create setsummary failed\n");
+ goto error_unlock_exit;
+ }
+ setsum->type = params->type;
+ setsum->socket_id = params->socket_id;
+ setsum->key_len = params->key_len;
+ setsum->num_set = params->num_set;
+ setsum->name = params->name;
+ setsum->prim_hash_seed = params->prim_hash_seed;
+ setsum->sec_hash_seed = params->sec_hash_seed;
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ ret = rte_member_create_ht(setsum, params);
+ break;
+ case RTE_MEMBER_TYPE_VBF:
+ ret = rte_member_create_vbf(setsum, params);
+ break;
+ default:
+ goto error_unlock_exit;
+ }
+ if (ret < 0)
+ goto error_unlock_exit;
+ RTE_LOG(DEBUG, MEMBER, "Creating a setsummary table with mode %u\n",
+ setsum->type);
+
+ te->data = (void *)setsum;
+ TAILQ_INSERT_TAIL(member_list, te, next);
+ rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+ return setsum;
+
+error_unlock_exit:
+ rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK);
+ rte_member_free(setsum);
+ return NULL;
+}
+
+
+int
+rte_member_add(const void *ss, const void *key, MEMBER_SET_TYPE set_id)
+{
+ const struct rte_member_setsum *setsum = ss;
+
+ if (setsum == NULL || key == NULL)
+ return -EINVAL;
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ return rte_member_add_ht(setsum, key, set_id);
+ case RTE_MEMBER_TYPE_VBF:
+ return rte_member_add_vbf(setsum, key, set_id);
+ default:
+ return -EINVAL;
+ }
+}
+
+
+int
+rte_member_lookup(const void *ss, const void *key,
+ MEMBER_SET_TYPE *set_id)
+{
+ const struct rte_member_setsum *setsum = ss;
+
+ if (setsum == NULL || key == NULL || set_id == NULL)
+ return -EINVAL;
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ return rte_member_lookup_ht(setsum, key, set_id);
+ case RTE_MEMBER_TYPE_VBF:
+ return rte_member_lookup_vbf(setsum, key, set_id);
+ default:
+ return -EINVAL;
+ }
+}
+
+
+int
+rte_member_lookup_bulk(const void *ss, const void **keys, uint32_t num_keys,
+ MEMBER_SET_TYPE *set_ids)
+{
+ const struct rte_member_setsum *setsum = ss;
+
+ if (setsum == NULL || keys == NULL || set_ids == NULL)
+ return -EINVAL;
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ return rte_member_lookup_bulk_ht(setsum, keys, num_keys,
+ set_ids);
+ case RTE_MEMBER_TYPE_VBF:
+ return rte_member_lookup_bulk_vbf(setsum, keys, num_keys,
+ set_ids);
+ default:
+ return -EINVAL;
+ }
+}
+
+int
+rte_member_lookup_multi(const void *ss, const void *key,
+ uint32_t match_per_key, MEMBER_SET_TYPE *set_id)
+{
+ const struct rte_member_setsum *setsum = ss;
+
+ if (setsum == NULL || key == NULL || set_id == NULL)
+ return -EINVAL;
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ return rte_member_lookup_multi_ht(setsum, key, match_per_key,
+ set_id);
+ case RTE_MEMBER_TYPE_VBF:
+ return rte_member_lookup_multi_vbf(setsum, key, match_per_key,
+ set_id);
+ default:
+ return -EINVAL;
+ }
+}
+
+
+int
+rte_member_lookup_multi_bulk(const void *ss, const void **keys,
+ uint32_t num_keys, uint32_t max_match_per_key,
+ uint32_t *match_count, MEMBER_SET_TYPE *set_ids)
+{
+ const struct rte_member_setsum *setsum = ss;
+
+ if (setsum == NULL || keys == NULL || set_ids == NULL ||
+ match_count == NULL)
+ return -EINVAL;
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ return rte_member_lookup_multi_bulk_ht(setsum, keys, num_keys,
+ max_match_per_key, match_count, set_ids);
+ case RTE_MEMBER_TYPE_VBF:
+ return rte_member_lookup_multi_bulk_vbf(setsum, keys, num_keys,
+ max_match_per_key, match_count, set_ids);
+ default:
+ return -EINVAL;
+ }
+}
+
+
+int
+rte_member_delete(void *ss, const void *key, MEMBER_SET_TYPE set_id)
+{
+ struct rte_member_setsum *setsum = ss;
+
+ if (setsum == NULL || key == NULL)
+ return -EINVAL;
+
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ return rte_member_delete_ht(setsum, key, set_id);
+ case RTE_MEMBER_TYPE_VBF:
+ default:
+ return -EINVAL;
+ }
+}
+
+
+void
+rte_member_reset(const void *ss)
+{
+ const struct rte_member_setsum *setsum = ss;
+
+ if (setsum == NULL)
+ return;
+ switch (setsum->type) {
+ case RTE_MEMBER_TYPE_HT:
+ rte_member_reset_ht(setsum);
+ return;
+ case RTE_MEMBER_TYPE_VBF:
+ rte_member_reset_vbf(setsum);
+ return;
+ default:
+ return;
+ }
+}
diff --git a/lib/librte_member/rte_member.h b/lib/librte_member/rte_member.h
new file mode 100644
index 0000000..de44b1b
--- /dev/null
+++ b/lib/librte_member/rte_member.h
@@ -0,0 +1,518 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+
+
+/**
+ * @file
+ *
+ * RTE Membership Library
+ *
+ * The Membership Library is an extension and generalization of a traditional
+ * filter (for example Bloom Filter) structure that has multiple usages in a
+ * variety of workloads and applications. The library is used to test if a key
+ * belongs to certain sets. Two types of such "set-summary" structures are
+ * implemented: hash-table based (HT) and vector bloom filter (vBF). For HT
+ * setsummary, two subtype or modes are available, cache and non-cache modes.
+ * The table below summarize some properties of the different implementations.
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ */
+
+
+/**
+ * <!--
+ * +==========+=====================+================+=========================+
+ * | type | vbf | HT-cache | HT-non-cache |
+ * +==========+=====================+==========================================+
+ * |structure | bloom-filter array | hash-table like without storing key |
+ * +----------+---------------------+------------------------------------------+
+ * |set id | limited by bf count | [1, 0x7fff] |
+ * | | up to 32. | |
+ * +----------+---------------------+------------------------------------------+
+ * |usages & | small set range, | can delete, | cache most recent keys, |
+ * |properties| user-specified | big set range, | have both false-positive|
+ * | | false-positive rate,| small false | and false-negative |
+ * | | no deletion support.| positive depend| depend on table size, |
+ * | | | on table size, | automatic overwritten. |
+ * | | | new key does | |
+ * | | | not overwrite | |
+ * | | | existing key. | |
+ * +----------+---------------------+----------------+-------------------------+
+ * -->
+ */
+
+
+#ifndef _RTE_MEMBER_H_
+#define _RTE_MEMBER_H_
+
+
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+#include <stdio.h>
+#include <stdint.h>
+#include <rte_hash_crc.h>
+
+/** The set ID type that stored internally in hash table based set summary. */
+typedef uint16_t MEMBER_SET_TYPE;
+/** Invalid set ID used to mean no match found. */
+#define RTE_MEMBER_NO_MATCH 0
+/** Maximum size of hash table that can be created. */
+#define RTE_MEMBER_ENTRIES_MAX (1 << 30)
+/** Maximum number of keys that can be searched as a bulk */
+#define RTE_MEMBER_LOOKUP_BULK_MAX 64
+/** Entry count per bucket in hash table based mode. */
+#define RTE_MEMBER_BUCKET_ENTRIES 16
+/** Maximum number of characters in setsum name. */
+#define RTE_MEMBER_NAMESIZE 32
+
+/** @internal Primary hash function to calculate 1st independent hash. */
+#define MEMBER_PRIM_HASH(key, key_len, seed) \
+ (uint32_t)(rte_hash_crc(key, key_len, seed))
+/** @internal Secondary hash function to calculate 2nd independent hash. */
+#define MEMBER_SEC_HASH(key, key_len, seed) \
+ (uint32_t)(rte_hash_crc(key, key_len, seed))
+
+
+struct rte_member_setsum;
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Parameter struct used to create set summary
+ */
+struct rte_member_parameters;
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Define different set summary types
+ */
+enum rte_member_setsum_type {
+ RTE_MEMBER_TYPE_HT = 0, /**< Hash table based set summary. */
+ RTE_MEMBER_TYPE_VBF, /**< vector of bloom filters. */
+ RTE_MEMBER_NUM_TYPE
+};
+
+/** @internal compare function for different arch. */
+enum rte_member_sig_compare_function {
+ RTE_MEMBER_COMPARE_SCALAR = 0,
+ RTE_MEMBER_COMPARE_AVX2,
+ RTE_MEMBER_COMPARE_NUM
+};
+
+/** @internal setsummary structure. */
+struct rte_member_setsum {
+ enum rte_member_setsum_type type;
+ const char *name;
+ uint32_t key_len;
+ uint32_t socket_id; /* NUMA Socket ID for memory. */
+ uint32_t prim_hash_seed;
+ uint32_t sec_hash_seed;
+
+
+ /* hash table based */
+ uint32_t bucket_cnt;
+ uint32_t bucket_mask;
+ uint8_t cache;
+ enum rte_member_sig_compare_function sig_cmp_fn;
+
+ /* vector bloom filter*/
+ uint32_t num_set;
+ uint32_t bits;
+ uint32_t bit_mask;
+ uint32_t num_hashes;
+ void *table;
+};
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Parameters used when create the set summary table. Currently user can
+ * specify two types of setsummary: HT based and vBF. For HT based, user can
+ * specify cache or non-cache mode. Here is a table to describe some differences
+ *
+ */
+struct rte_member_parameters {
+ const char *name; /**< Name of the hash. */
+
+ /**
+ * User to specify the type of the setsummary from one of
+ * rte_member_setsum_type.
+ *
+ * HT based setsummary is implemented like a hash table. User should use
+ * this type when there are many sets.
+ *
+ * vBF setsummary is a vector of bloom filters. It is used when number
+ * of sets is not big (less than 32 for current implementation).
+ */
+ enum rte_member_setsum_type type;
+ /**
+ * If it is HT based setsummary, user to specify the subtype or mode
+ * of the setsummary. It could be cache, or non-cache mode.
+ * Set iscache to be 1 if to use as cache mode.
+ *
+ * For cache mode, keys can be evicted out of the HT setsummary. Keys
+ * with the same signature and map to the same bucket
+ * will overwrite each other in the setsummary table.
+ * This mode is useful for the case that the set-summary only
+ * needs to keep record of the recently inserted keys. Both
+ * false-negative and false-positive could happen.
+ *
+ * For non-cache mode, keys cannot be evicted out of the cache. So for
+ * this mode the setsummary will become full eventually. Keys with the
+ * same signature but map to the same bucket will still occupy multiple
+ * entries. This mode does not give false-negative result.
+ */
+ uint8_t iscache;
+
+ /**
+ * For HT setsummary, num_keys equals to the number of entries of the
+ * table. When the number of keys that inserted to the HT setsummary
+ * approaches this number, eviction could happen. For cache mode,
+ * keys could be evicted out of the table. For non-cache mode, keys will
+ * be evicted to other buckets like cuckoo hash. The table will also
+ * likely to become full before the number of inserted keys equal to the
+ * total number of entries.
+ *
+ * For vBF, num_keys equal to the expected number of keys that will
+ * be inserted into the vBF. The implementation assumes the keys are
+ * evenly distributed to each BF in vBF. This is used to calculate the
+ * number of bits we need for each BF. User does not specify the size of
+ * each BF directly because the optimal size depends on the num_keys
+ * and false positive rate.
+ */
+ uint32_t num_keys;
+
+
+ /**
+ * The length of key is used for hash calculation. Since key is not
+ * stored in set-summary, large key does not require more memory space.
+ */
+ uint32_t key_len;
+
+
+ /**
+ * num_set is only relevant to vBF based setsummary.
+ * num_set is equal to the number of BFs in vBF. For current
+ * implementation, it only supports 1,2,4,8,16,32 BFs in one vBF set
+ * summary. If other number of sets are needed, for example 5, the user
+ * should allocate the minimum available value that larger than 5,
+ * which is 8.
+ */
+ uint32_t num_set;
+
+ /**
+ * false_postive_rate is only relevant to vBF based setsummary.
+ * false_postivie_rate is the user-defined false positive rate
+ * given expected number of inserted keys (num_keys). It is used to
+ * calculate the total number of bits for each BF, and the number of
+ * hash values used during lookup and insertion. For details please
+ * refer to vBF implementation and membership library documentation.
+ *
+ * HT setsummary's false positive rate is in the order of:
+ * false_pos = (1/bucket_count)*(1/2^16), since we use 16-bit signature.
+ * This is because two keys needs to map to same bucket and same
+ * signature to have a collision (false positive). bucket_count is equal
+ * to number of entries (num_keys) divided by entry count per bucket
+ * (RTE_MEMBER_BUCKET_ENTRIES). Thus, the false_postivie_rate is not
+ * directly set by users.
+ */
+ float false_positive_rate;
+
+ /**
+ * We use two seeds to calculate two independent hashes for each key.
+ *
+ * For HT type, one hash is used as signature, and the other is used
+ * for bucket location.
+ * For vBF type, these two hashes and their combinations are used as
+ * hash locations to index the bit array.
+ */
+ uint32_t prim_hash_seed;
+
+ /**
+ * The secondary seed should be a different value from the primary seed.
+ */
+ uint32_t sec_hash_seed;
+
+ int socket_id; /**< NUMA Socket ID for memory. */
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Find an existing set-summary and return a pointer to it.
+ *
+ * @param name
+ * Name of the set-summary
+ * @return
+ * Pointer to the set-summary or NULL if object not found
+ * with rte_errno set appropriately. Possible rte_errno values include:
+ * - ENOENT - value not available for return
+ */
+void *
+rte_member_find_existing(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Create set-summary (SS).
+ *
+ * @param params
+ * parameters to initialize the setsummary
+ * @return
+ * return the pointer to the setsummary
+ * return value is NULL if the creation failed
+ */
+
+void *
+rte_member_create(const struct rte_member_parameters *params);
+
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Lookup key in set-summary (SS).
+ * Single key lookup and return as soon as the first match found
+ * @param setsum
+ * pointer of a setsummary
+ * @param key
+ * pointer of the key that needs to lookup
+ * @param set_id
+ * output the set id matches the key
+ * @return
+ * return 1 for found a match and 0 for not found a match
+ */
+
+int
+rte_member_lookup(const void *setsum,
+ const void *key, MEMBER_SET_TYPE *set_id);
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * lookup bulk of keys in set-summary (SS).
+ * Each key lookup returns as soon as the first match found
+ * @param setsum
+ * Pointer of a setsummary
+ * @param keys
+ * Pointer of bulk of keys that to be lookup
+ * @param num_keys
+ * Number of keys that will be lookup
+ * @param set_ids
+ * Output set ids for all the keys to this array.
+ * User should preallocate array that can contain all results, which size is
+ * the num_keys.
+ * @return
+ * The number of keys that found a match
+ */
+
+
+int
+rte_member_lookup_bulk(const void *setsum,
+ const void **keys, uint32_t num_keys,
+ MEMBER_SET_TYPE *set_ids);
+
+
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Lookup a key in set-summary (SS) for multiple matches.
+ * The key lookup will find all matched entries (multiple match).
+ * Note that for cache mode of HT, each key can have at most one match. This is
+ * because keys with same signature that maps to same bucket will overwrite
+ * each other. So multi-match lookup should be used for vBF and non-cache HT.
+ * @param setsum
+ * pointer of a set-summary
+ * @param key
+ * The key that to be lookup
+ * @param max_match_per_key
+ * User specified maximum number of matches for each key. The function returns
+ * as soon as this number of matches found for the key.
+ * @param set_id
+ * Output set ids for all the matches of the key. User needs to preallocate
+ * the array that can contain max_match_per_key number of results.
+ * @return
+ * The number of matches that found for the key.
+ * For cache mode HT set-summary, the number should be at most 1
+ */
+
+int
+rte_member_lookup_multi(const void *setsum,
+ const void *key, uint32_t max_match_per_key,
+ MEMBER_SET_TYPE *set_id);
+
+
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Lookup a bulk of keys in set-summary (SS) for multiple matches each key.
+ * Each key lookup will find all matched entries (multiple match).
+ * Note that for cache mode HT, each key can have at most one match. So
+ * multi-match function is mainly used for vBF and non-cache mode HT.
+ * @param setsum
+ * pointer of a setsummary
+ * @param keys
+ * The keys that to be lookup
+ * @param num_keys
+ * The number of keys that will be lookup
+ * @param max_match_per_key
+ * The possible maximum number of matches for each key
+ * @param match_count
+ * Output the number of matches for each key in an array
+ * @param set_ids
+ * Return set ids for all the matches of all keys. User pass in a preallocated
+ * 2D array with first dimension as key index and second dimension as match
+ * index. For example set_ids[bulk_size][max_match_per_key]
+ * @return
+ * The number of keys that found one or more matches in the set-summary
+ */
+
+int
+rte_member_lookup_multi_bulk(const void *setsum,
+ const void **keys, uint32_t num_keys,
+ uint32_t max_match_per_key,
+ uint32_t *match_count,
+ MEMBER_SET_TYPE *set_ids);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Insert key into set-summary (SS).
+ *
+ * @param setsum
+ * pointer of a set-summary
+ * @param key
+ * the key that needs to be added
+ * @param set_id
+ * The set id associated with the key that needs to be added. Different mode
+ * supports different set_id ranges. 0 cannot be used as set_id since
+ * RTE_MEMBER_NO_MATCH by default is set as 0.
+ * For HT mode, the set_id has range as [1, 0x7FFF], MSB is reserved.
+ * For vBF mode the set id is limited by the num_set parameter when create
+ * the set-summary.
+ * @return
+ * HT (cache mode) and vBF should never fail unless the set_id is not in the
+ * valid range. In such case -EINVAL is returned.
+ * For HT (non-cache mode) it could fail with -ENOSPC error code when table is
+ * full.
+ * For success it returns different values for different modes to provide
+ * extra information for users.
+ * Return 0 for HT (cache mode) if the add does not cause
+ * eviction, return 1 otherwise. Return 0 for HT mode if success, -ENOSPC for
+ * full, and 1 if cuckoo eviction happens. Always return 0 for vBF mode.
+ */
+
+int
+rte_member_add(const void *setsum, const void *key,
+ MEMBER_SET_TYPE set_id);
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * De-allocate memory used by set-summary.
+ * @param setsum
+ * Pointer to the set summary
+ */
+void
+rte_member_free(void *setsum);
+
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Reset the set-summary tables. E.g. reset bits to be 0 in BF,
+ * reset set_id in each entry to be RTE_MEMBER_NO_MATCH in HT based SS.
+ * @param setsum
+ * Pointer to the set-summary
+ */
+void
+rte_member_reset(const void *setsum);
+
+
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Delete items from the set-summary. Note that vBF does not support deletion
+ * in current implementation. For vBF, error code of -EINVAL will be returned.
+ * @param setsum
+ * Pointer to the set-summary
+ * @param key
+ * The key to be deleted
+ * @param set_id
+ * For HT mode, we need both key and its corresponding set_id to
+ * properly delete the key. Without set_id, we may delete other keys with the
+ * same signature
+ * @return
+ * If no entry found to delete, an error code of -ENOENT could be returned
+ */
+
+int
+rte_member_delete(void *setsum, const void *key,
+ MEMBER_SET_TYPE set_id);
+
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_MEMBER_H_ */
diff --git a/lib/librte_member/rte_member_version.map b/lib/librte_member/rte_member_version.map
new file mode 100644
index 0000000..a5877c9
--- /dev/null
+++ b/lib/librte_member/rte_member_version.map
@@ -0,0 +1,16 @@
+DPDK_17.11 {
+ global:
+
+ rte_member_create;
+ rte_member_find_existing;
+ rte_member_lookup;
+ rte_member_lookup_bulk;
+ rte_member_lookup_multi;
+ rte_member_lookup_multi_bulk;
+ rte_member_add;
+ rte_member_free;
+ rte_member_reset;
+ rte_member_delete;
+
+ local: *;
+};
--
2.7.4
Thomas Monjalon
2017-09-22 10:47:18 UTC
Permalink
Post by Yipeng Wang
--- a/lib/librte_eal/common/eal_common_log.c
+++ b/lib/librte_eal/common/eal_common_log.c
@@ -279,6 +279,7 @@ static const struct logtype logtype_strings[] = {
{RTE_LOGTYPE_CRYPTODEV, "cryptodev"},
{RTE_LOGTYPE_EFD, "efd"},
{RTE_LOGTYPE_EVENTDEV, "eventdev"},
+ {RTE_LOGTYPE_MEMBER, "member"},
{RTE_LOGTYPE_USER1, "user1"},
{RTE_LOGTYPE_USER2, "user2"},
{RTE_LOGTYPE_USER3, "user3"},
The static logtypes should be removed.
Please use the new dynamic log types.
De Lara Guarch, Pablo
2017-09-25 14:15:03 UTC
Permalink
-----Original Message-----
Sent: Wednesday, September 6, 2017 1:00 AM
Subject: [dpdk-dev] [PATCH v3 1/7] member: implement main API
Membership library is an extension and generalization of a traditional filter
(for example Bloom Filter) structure. In general, the Membership library is a
data structure that provides a "set-summary" and responds to set-
membership queries of whether a certain element belongs to a set(s). A
membership test for an element will return the set this element belongs to
or not-found if the element is never inserted into the set-summary.
The results of the membership test is not 100% accurate. Certain false
Is -> are.
positive or false negative probability could exist. However, comparing to a
"full-blown" complete list of elements, a "set-summary"
is memory efficient and fast on lookup.
This patch add the main API definition.
---
lib/Makefile | 2 +
lib/librte_eal/common/eal_common_log.c | 1 +
lib/librte_eal/common/include/rte_log.h | 1 +
lib/librte_member/Makefile | 49 +++
lib/librte_member/rte_member.c | 342 ++++++++++++++++++++
lib/librte_member/rte_member.h | 518
+++++++++++++++++++++++++++++++
lib/librte_member/rte_member_version.map | 16 +
7 files changed, 929 insertions(+)
create mode 100644 lib/librte_member/Makefile create mode 100644
lib/librte_member/rte_member.c create mode 100644
lib/librte_member/rte_member.h create mode 100644
lib/librte_member/rte_member_version.map
...
diff --git a/lib/librte_member/rte_member.c
b/lib/librte_member/rte_member.c new file mode 100644 index
0000000..71b066d
--- /dev/null
+++ b/lib/librte_member/rte_member.c
...
+
+#include <string.h>
+
+#include <rte_eal.h>
+#include <rte_eal_memconfig.h>
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_malloc.h>
+#include <rte_errno.h>
+
+#include "rte_member.h"
+#include "rte_member_ht.h"
+#include "rte_member_vbf.h"
+
+TAILQ_HEAD(rte_member_list, rte_tailq_entry); static struct
+rte_tailq_elem rte_member_tailq = {
+ .name = "RTE_MEMBER",
+};
+EAL_REGISTER_TAILQ(rte_member_tailq)
+
+
+void *
+rte_member_find_existing(const char *name) {
+ struct rte_member_setsum *setsum = NULL;
+ struct rte_tailq_entry *te;
+ struct rte_member_list *member_list;
+
+ member_list = RTE_TAILQ_CAST(rte_member_tailq.head,
rte_member_list);
+
+ rte_rwlock_read_lock(RTE_EAL_TAILQ_RWLOCK);
+ TAILQ_FOREACH(te, member_list, next) {
+ setsum = (struct rte_member_setsum *) te->data;
+ if (strncmp(name, setsum->name,
RTE_MEMBER_NAMESIZE) == 0)
+ break;
+ }
+ rte_rwlock_read_unlock(RTE_EAL_TAILQ_RWLOCK);
+
+ if (te == NULL) {
+ rte_errno = ENOENT;
+ return NULL;
+ }
+ return setsum;
+}
+
+void
+rte_member_free(void *ss)
Why not using directly "struct rte_member_setsum", so you can use it directly?
This applies to other functions that are using "void *".
I see that the content of this structure is internal, but you can still use a pointer to that structure.

...
+void *
+rte_member_create(const struct rte_member_parameters *params) {
+ struct rte_tailq_entry *te;
+ struct rte_member_list *member_list = NULL;
+ struct rte_member_setsum *setsum = NULL;
+ int ret;
+
+ if (params == NULL) {
+ rte_errno = EINVAL;
+ return NULL;
+ }
+
+ if ((params->key_len == 0)) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, MEMBER,
+ "Memship create with invalid parameters\n");
rte_member_create has invalid parameters?
diff --git a/lib/librte_member/rte_member.h
b/lib/librte_member/rte_member.h new file mode 100644 index
0000000..de44b1b
--- /dev/null
+++ b/lib/librte_member/rte_member.h
@@ -0,0 +1,518 @@
...
+enum rte_member_setsum_type {
+ RTE_MEMBER_TYPE_HT = 0, /**< Hash table based set summary.
*/
+ RTE_MEMBER_TYPE_VBF, /**< vector of bloom filters. */
"vector" -> "Vector".
+ RTE_MEMBER_NUM_TYPE
+};
+
+rte_member_sig_compare_function {
+ RTE_MEMBER_COMPARE_SCALAR = 0,
+ RTE_MEMBER_COMPARE_AVX2,
+ RTE_MEMBER_COMPARE_NUM
+};
+
+struct rte_member_setsum {
+ enum rte_member_setsum_type type;
+ const char *name;
+ uint32_t key_len;
+ uint32_t socket_id; /* NUMA Socket ID for memory. */
Either comment all the fields or do not do it (for this case, because the
structure is internal, otherwise, all fields would require a comment).
Also, remember that they should all start with capital letters.
+ uint32_t prim_hash_seed;
+ uint32_t sec_hash_seed;
+
+
+ /* hash table based */
+ uint32_t bucket_cnt;
+ uint32_t bucket_mask;
+ uint8_t cache;
+ enum rte_member_sig_compare_function sig_cmp_fn;
...
+struct rte_member_parameters {
...
+ uint8_t iscache;
As said in the documentation patch, change to "is_cache".
+
+ /**
+ * For HT setsummary, num_keys equals to the number of entries of the
+ * table. When the number of keys that inserted to the HT
setsummary
"number of keys inserted in the HT summary".
+ * approaches this number, eviction could happen. For cache mode,
+ * keys could be evicted out of the table. For non-cache mode, keys will
+ * be evicted to other buckets like cuckoo hash. The table will also
+ * likely to become full before the number of inserted keys equal to the
+ * total number of entries.
+ *
+ * For vBF, num_keys equal to the expected number of keys that will
+ * be inserted into the vBF. The implementation assumes the keys are
+ * evenly distributed to each BF in vBF. This is used to calculate the
+ * number of bits we need for each BF. User does not specify the size of
+ * each BF directly because the optimal size depends on the
num_keys
+ * and false positive rate.
+ */
+ uint32_t num_keys;
+
+
Leave just one blank line between fields.
+ /**
+ * The length of key is used for hash calculation. Since key is not
+ * stored in set-summary, large key does not require more memory space.
+ */
+ uint32_t key_len;
+
+
+ /**
+ * num_set is only relevant to vBF based setsummary.
+ * num_set is equal to the number of BFs in vBF. For current
+ * implementation, it only supports 1,2,4,8,16,32 BFs in one vBF set
+ * summary. If other number of sets are needed, for example 5, the user
+ * should allocate the minimum available value that larger than 5,
+ * which is 8.
+ */
+ uint32_t num_set;
+
+ /**
+ * false_postive_rate is only relevant to vBF based setsummary.
+ * false_postivie_rate is the user-defined false positive rate
+ * given expected number of inserted keys (num_keys). It is used to
+ * calculate the total number of bits for each BF, and the number of
+ * hash values used during lookup and insertion. For details please
+ * refer to vBF implementation and membership library
documentation.
+ *
+ * false_pos = (1/bucket_count)*(1/2^16), since we use 16-bit signature.
+ * This is because two keys needs to map to same bucket and same
+ * signature to have a collision (false positive). bucket_count is equal
+ * to number of entries (num_keys) divided by entry count per bucket
+ * (RTE_MEMBER_BUCKET_ENTRIES). Thus, the false_postivie_rate is not
+ * directly set by users.
+ */
Typo in "positive" in several places in the comment above.
Also, clarify that "false_positive_rate" is not directly set by users for HT mode.
+ float false_positive_rate;
...
+ * Lookup key in set-summary (SS).
+ * Single key lookup and return as soon as the first match found
Leave a blank line before parameters.
+ * pointer of a setsummary
+ * pointer of the key that needs to lookup
Better something like this "Pointer of the key to be looked up"
+ * output the set id matches the key
+ * return 1 for found a match and 0 for not found a match
+ */
+
Definitions of the fields should start with capital letter.
+int
+rte_member_lookup(const void *setsum,
+ const void *key, MEMBER_SET_TYPE *set_id);
+
+
+/**
+ *
+ * lookup bulk of keys in set-summary (SS).
"Lookup bulk"
+ * Each key lookup returns as soon as the first match found
+ * Pointer of a setsummary
+ * Pointer of bulk of keys that to be lookup
"Pointer of bulk of keys to be looked up".
Check for this in other parts of the code.
+ * Number of keys that will be lookup
+ * Output set ids for all the keys to this array.
+ * User should preallocate array that can contain all results, which size is
+ * the num_keys.
+ * The number of keys that found a match
+ */
+
+
+int
+rte_member_lookup_bulk(const void *setsum,
+ const void **keys, uint32_t num_keys,
+ MEMBER_SET_TYPE *set_ids);
+
+
+
+
Too many blank spaces. Generally leave one between functions.
Check for this in the rest of the files.
+/**
+ *
+ * Lookup a bulk of keys in set-summary (SS) for multiple matches each key.
+ * Each key lookup will find all matched entries (multiple match).
+ * Note that for cache mode HT, each key can have at most one match. So
+ * multi-match function is mainly used for vBF and non-cache mode HT.
+ * pointer of a setsummary
+ * The keys that to be lookup
+ * The number of keys that will be lookup
+ * The possible maximum number of matches for each key
+ * Output the number of matches for each key in an array
+ * Return set ids for all the matches of all keys. User pass in a preallocated
"User passes in"
+ * 2D array with first dimension as key index and second dimension as match
+ * index. For example set_ids[bulk_size][max_match_per_key]
+ * The number of keys that found one or more matches in the set-
summary
+ */
+
+int
+rte_member_lookup_multi_bulk(const void *setsum,
+ const void **keys, uint32_t num_keys,
+ uint32_t max_match_per_key,
+ uint32_t *match_count,
+ MEMBER_SET_TYPE *set_ids);
+
+/**
+ *
+ * Insert key into set-summary (SS).
+ *
+ * pointer of a set-summary
Start with capital letter.
+ * the key that needs to be added
+ * The set id associated with the key that needs to be added. Different mode
+ * supports different set_id ranges. 0 cannot be used as set_id since
+ * RTE_MEMBER_NO_MATCH by default is set as 0.
+ * For HT mode, the set_id has range as [1, 0x7FFF], MSB is reserved.
+ * For vBF mode the set id is limited by the num_set parameter when create
+ * the set-summary.
+ * HT (cache mode) and vBF should never fail unless the set_id is not in the
+ * valid range. In such case -EINVAL is returned.
+ * For HT (non-cache mode) it could fail with -ENOSPC error code when table is
+ * full.
+ * For success it returns different values for different modes to provide
+ * extra information for users.
+ * Return 0 for HT (cache mode) if the add does not cause
+ * eviction, return 1 otherwise. Return 0 for HT mode if success, -ENOSPC
For HT non-cache mode?
for
+ * full, and 1 if cuckoo eviction happens. Always return 0 for vBF mode.
+ */
+
+int
+rte_member_add(const void *setsum, const void *key,
+ MEMBER_SET_TYPE set_id);
+
+
...
diff --git a/lib/librte_member/rte_member_version.map
b/lib/librte_member/rte_member_version.map
new file mode 100644
index 0000000..a5877c9
--- /dev/null
+++ b/lib/librte_member/rte_member_version.map
@@ -0,0 +1,16 @@
+DPDK_17.11 {
+
+ rte_member_create;
+ rte_member_find_existing;
+ rte_member_lookup;
+ rte_member_lookup_bulk;
+ rte_member_lookup_multi;
+ rte_member_lookup_multi_bulk;
+ rte_member_add;
+ rte_member_free;
+ rte_member_reset;
+ rte_member_delete;
This list must be in alphabetical order.
+
+ local: *;
+};
--
2.7.4
Yipeng Wang
2017-09-05 23:59:44 UTC
Permalink
One of the set-summray structure is hash-table based
set-summary (HTSS). One example is cuckoo filter [1].

Comparing to a traditional hash table, HTSS has a much more
compact structure. For each element, only one signature and
its corresponding set ID is stored. No key comparison is required
during lookup. For the table structure, there are multiple entries
in each bucket, and the table is composed of many buckets.

Two modes are supported for HTSS, "cache" and "none-cache" modes.
The non-cache mode is more similar to traditional cuckoo filter.
When a bucket is full, one entry will be evicted to its
alternative bucket to make space for the new key. The table could
be full and then no more keys could be inserted. This mode has
false-positive rate but no false-negative. Multiple entries
with same signature could stay in the same bucket.

The "cache" mode does not evict key to its alternative bucket
when a bucket is full, an existing key will be evicted out of
the table like a cache. So the table will never reject keys when
it is full. Another property is in each bucket, there cannot be
multiple entries with same signature. The mode could have both
false-positive and false-negative probability.

This patch add the implementation of HTSS.

[1] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically
Better Than Bloom,” in Conference on emerging Networking
Experiments and Technologies, 2014.

Signed-off-by: Yipeng Wang <***@intel.com>
---
lib/librte_member/Makefile | 2 +-
lib/librte_member/rte_member_ht.c | 490 ++++++++++++++++++++++++++++++++++++++
lib/librte_member/rte_member_ht.h | 115 +++++++++
3 files changed, 606 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_member/rte_member_ht.c
create mode 100644 lib/librte_member/rte_member_ht.h

diff --git a/lib/librte_member/Makefile b/lib/librte_member/Makefile
index 1a79eaa..ad26548 100644
--- a/lib/librte_member/Makefile
+++ b/lib/librte_member/Makefile
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map
LIBABIVER := 1

# all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c
# install includes
SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h

diff --git a/lib/librte_member/rte_member_ht.c b/lib/librte_member/rte_member_ht.c
new file mode 100644
index 0000000..b2ae6d0
--- /dev/null
+++ b/lib/librte_member/rte_member_ht.c
@@ -0,0 +1,490 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <rte_errno.h>
+#include <rte_malloc.h>
+#include <rte_prefetch.h>
+#include <rte_random.h>
+#include <rte_log.h>
+
+#include "rte_member.h"
+#include "rte_member_ht.h"
+
+
+static inline int
+insert_overwrite_search(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ MEMBER_SET_TYPE set_id)
+{
+ int i;
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ if (buckets[bucket].sigs[i] == tmp_sig) {
+ buckets[bucket].sets[i] = set_id;
+ return 1;
+ }
+ }
+ return 0;
+}
+
+
+
+
+static inline int
+search_bucket_single(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ MEMBER_SET_TYPE *set_id)
+{
+ int iter;
+ for (iter = 0; iter < RTE_MEMBER_BUCKET_ENTRIES; iter++) {
+ if (tmp_sig == buckets[bucket].sigs[iter] &&
+ buckets[bucket].sets[iter] !=
+ RTE_MEMBER_NO_MATCH) {
+ *set_id = buckets[bucket].sets[iter];
+ return 1;
+ }
+ }
+ return 0;
+}
+
+
+static inline void
+search_bucket_multi(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ uint32_t *counter,
+ uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id)
+{
+ int iter;
+ for (iter = 0; iter < RTE_MEMBER_BUCKET_ENTRIES; iter++) {
+ if (tmp_sig == buckets[bucket].sigs[iter] &&
+ buckets[bucket].sets[iter] !=
+ RTE_MEMBER_NO_MATCH) {
+ set_id[*counter] = buckets[bucket].sets[iter];
+ (*counter)++;
+ if (*counter >= match_per_key)
+ return;
+ }
+ }
+}
+
+
+int
+rte_member_create_ht(struct rte_member_setsum *ss,
+ const struct rte_member_parameters *params)
+{
+ uint32_t i, j;
+ uint32_t size_bucket_t;
+ uint32_t num_entries = rte_align32pow2(params->num_keys);
+
+ if ((num_entries > RTE_MEMBER_ENTRIES_MAX) ||
+ !rte_is_power_of_2(RTE_MEMBER_BUCKET_ENTRIES) ||
+ num_entries < RTE_MEMBER_BUCKET_ENTRIES) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, MEMBER,
+ "Membership HT create with invalid parameters\n");
+ return -EINVAL;
+ }
+
+ uint32_t num_buckets = num_entries / RTE_MEMBER_BUCKET_ENTRIES;
+
+ size_bucket_t = sizeof(struct member_ht_bucket);
+
+ struct member_ht_bucket *buckets = rte_zmalloc_socket(NULL,
+ num_buckets * size_bucket_t,
+ RTE_CACHE_LINE_SIZE, ss->socket_id);
+
+ if (buckets == NULL)
+ return -ENOMEM;
+
+ ss->table = buckets;
+ ss->bucket_cnt = num_buckets;
+ ss->bucket_mask = num_buckets - 1;
+ ss->cache = params->iscache;
+
+ for (i = 0; i < num_buckets; i++) {
+ for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++)
+ buckets[i].sets[j] = RTE_MEMBER_NO_MATCH;
+ }
+
+
+ RTE_LOG(DEBUG, MEMBER, "Hash table based filter created, "
+ "the table has %u entries, %u buckets\n",
+ num_buckets,
+ num_buckets / RTE_MEMBER_BUCKET_ENTRIES);
+ return 0;
+}
+
+static inline
+void get_buckets_index(const struct rte_member_setsum *ss, const void *key,
+ uint32_t *prim_bkt, uint32_t *sec_bkt, SIG_TYPE *sig)
+{
+ uint32_t first_hash = MEMBER_PRIM_HASH(key, ss->key_len,
+ ss->prim_hash_seed);
+ uint32_t sec_hash = MEMBER_SEC_HASH(&first_hash, 4, ss->sec_hash_seed);
+ *sig = first_hash & SIG_BITMASK;
+ if (ss->cache) {
+ *prim_bkt = sec_hash & ss->bucket_mask;
+ *sec_bkt = (sec_hash >> 16) & ss->bucket_mask;
+ } else {
+ *prim_bkt = sec_hash & ss->bucket_mask;
+ *sec_bkt = (*prim_bkt ^ *sig) & ss->bucket_mask;
+ }
+}
+
+
+int
+rte_member_lookup_ht(const struct rte_member_setsum *ss,
+ const void *key, MEMBER_SET_TYPE *set_id)
+{
+ uint32_t prim_bucket, sec_bucket;
+ SIG_TYPE tmp_sig;
+ struct member_ht_bucket *buckets = ss->table;
+
+
+ *set_id = RTE_MEMBER_NO_MATCH;
+ get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
+
+ if (search_bucket_single(prim_bucket, tmp_sig, buckets,
+ set_id) ||
+ search_bucket_single(sec_bucket, tmp_sig,
+ buckets, set_id))
+ return 1;
+
+ return 0;
+}
+
+
+uint32_t
+rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss,
+ const void **keys, uint32_t num_keys, MEMBER_SET_TYPE *set_id)
+{
+ uint32_t i;
+ uint32_t ret = 0;
+ struct member_ht_bucket *buckets = ss->table;
+ SIG_TYPE tmp_sig[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+ uint32_t prim_buckets[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+ uint32_t sec_buckets[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+
+ for (i = 0; i < num_keys; i++) {
+ get_buckets_index(ss, keys[i], &prim_buckets[i],
+ &sec_buckets[i], &tmp_sig[i]);
+ rte_prefetch0(&buckets[prim_buckets[i]]);
+ rte_prefetch0(&buckets[sec_buckets[i]]);
+ }
+
+ for (i = 0; i < num_keys; i++) {
+ if (search_bucket_single(prim_buckets[i], tmp_sig[i],
+ buckets, &set_id[i]) ||
+ search_bucket_single(sec_buckets[i],
+ tmp_sig[i], buckets, &set_id[i]))
+ ret++;
+ else
+ set_id[i] = RTE_MEMBER_NO_MATCH;
+ }
+ return ret;
+}
+
+
+uint32_t
+rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,
+ const void *key, uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id)
+{
+ uint32_t ret = 0;
+ uint32_t prim_bucket, sec_bucket;
+ SIG_TYPE tmp_sig;
+ struct member_ht_bucket *buckets = ss->table;
+
+ get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
+
+ search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
+ match_per_key, set_id);
+ if (ret < match_per_key)
+ search_bucket_multi(sec_bucket, tmp_sig,
+ buckets, &ret, match_per_key, set_id);
+ return ret;
+}
+
+
+uint32_t
+rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,
+ const void **keys, uint32_t num_keys, uint32_t match_per_key,
+ uint32_t *match_count,
+ MEMBER_SET_TYPE *set_ids)
+{
+ uint32_t i;
+ uint32_t ret = 0;
+ struct member_ht_bucket *buckets = ss->table;
+ uint32_t match_cnt_t;
+ SIG_TYPE tmp_sig[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+ uint32_t prim_buckets[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+ uint32_t sec_buckets[RTE_MEMBER_LOOKUP_BULK_MAX] = {0};
+
+ for (i = 0; i < num_keys; i++) {
+ get_buckets_index(ss, keys[i], &prim_buckets[i],
+ &sec_buckets[i], &tmp_sig[i]);
+ rte_prefetch0(&buckets[prim_buckets[i]]);
+ rte_prefetch0(&buckets[sec_buckets[i]]);
+ }
+ for (i = 0; i < num_keys; i++) {
+ match_cnt_t = 0;
+
+ search_bucket_multi(prim_buckets[i], tmp_sig[i],
+ buckets, &match_cnt_t, match_per_key,
+ &set_ids[i*match_per_key]);
+ if (match_cnt_t < match_per_key)
+ search_bucket_multi(sec_buckets[i], tmp_sig[i],
+ buckets, &match_cnt_t, match_per_key,
+ &set_ids[i*match_per_key]);
+ match_count[i] = match_cnt_t;
+ if (match_cnt_t != 0)
+ ret++;
+ }
+ return ret;
+}
+
+static inline int
+try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
+ SIG_TYPE sig, MEMBER_SET_TYPE set_id)
+{
+ int i;
+ /* If not full then insert into one slot*/
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ if (buckets[prim].sets[i] == RTE_MEMBER_NO_MATCH) {
+ buckets[prim].sigs[i] = sig;
+ buckets[prim].sets[i] = set_id;
+ return 0;
+ }
+ }
+ /* if prim failed, we need to access second cache line */
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ if (buckets[sec].sets[i] == RTE_MEMBER_NO_MATCH) {
+ buckets[sec].sigs[i] = sig;
+ buckets[sec].sets[i] = set_id;
+ return 0;
+ }
+ }
+ return -1;
+}
+
+
+static inline int
+try_overwrite(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
+ SIG_TYPE sig, MEMBER_SET_TYPE set_id)
+{
+ if (insert_overwrite_search(prim, sig, buckets, set_id) ||
+ insert_overwrite_search(sec, sig, buckets,
+ set_id))
+ return 0;
+ return -1;
+}
+
+static inline int
+evict_from_bucket(void)
+{
+ /* for now, we randomly pick one entry to evict */
+ return rte_rand() & (RTE_MEMBER_BUCKET_ENTRIES - 1);
+}
+
+
+
+/*
+ * This function is similar to the cuckoo hash make_space function in hash
+ * library
+ */
+static inline int
+make_space_bucket(const struct rte_member_setsum *ss, uint32_t bkt_num)
+{
+
+ static unsigned int nr_pushes;
+
+ unsigned int i, j;
+ int ret;
+ struct member_ht_bucket *buckets = ss->table;
+ uint32_t next_bucket_idx;
+ struct member_ht_bucket *next_bkt[RTE_MEMBER_BUCKET_ENTRIES];
+ struct member_ht_bucket *bkt = &buckets[bkt_num];
+ MEMBER_SET_TYPE flag_mask = 1U << (sizeof(MEMBER_SET_TYPE) * 8 - 1);
+ /*
+ * Push existing item (search for bucket with space in
+ * alternative locations) to its alternative location
+ */
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ /* Search for space in alternative locations */
+ next_bucket_idx = (bkt->sigs[i] ^ bkt_num) & ss->bucket_mask;
+ next_bkt[i] = &buckets[next_bucket_idx];
+ for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++) {
+ if (next_bkt[i]->sets[j] == RTE_MEMBER_NO_MATCH)
+ break;
+ }
+
+ if (j != RTE_MEMBER_BUCKET_ENTRIES)
+ break;
+ }
+
+ /* Alternative location has spare room (end of recursive function) */
+ if (i != RTE_MEMBER_BUCKET_ENTRIES) {
+ next_bkt[i]->sigs[j] = bkt->sigs[i];
+ next_bkt[i]->sets[j] = bkt->sets[i];
+ return i;
+ }
+
+ /* Pick entry that has not been pushed yet */
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++)
+ if ((bkt->sets[i] & flag_mask) == 0)
+ break;
+
+ /* All entries have been pushed, so entry cannot be added */
+ if (i == RTE_MEMBER_BUCKET_ENTRIES ||
+ nr_pushes > RTE_MEMBER_MAX_PUSHES)
+ return -ENOSPC;
+
+ next_bucket_idx = (bkt->sigs[i] ^ bkt_num) & ss->bucket_mask;
+ /* Set flag to indicate that this entry is going to be pushed */
+ bkt->sets[i] |= flag_mask;
+
+ nr_pushes++;
+ /* Need room in alternative bucket to insert the pushed entry */
+ ret = make_space_bucket(ss, next_bucket_idx);
+ /*
+ * After recursive function.
+ * Clear flags and insert the pushed entry
+ * in its alternative location if successful,
+ * or return error
+ */
+ bkt->sets[i] &= ~flag_mask;
+ nr_pushes = 0;
+ if (ret >= 0) {
+ next_bkt[i]->sigs[ret] = bkt->sigs[i];
+ next_bkt[i]->sets[ret] = bkt->sets[i];
+ return i;
+ } else
+ return ret;
+}
+
+
+int
+rte_member_add_ht(const struct rte_member_setsum *ss,
+ const void *key, MEMBER_SET_TYPE set_id)
+{
+ int ret;
+ uint32_t prim_bucket, sec_bucket;
+ SIG_TYPE tmp_sig;
+ struct member_ht_bucket *buckets = ss->table;
+ MEMBER_SET_TYPE flag_mask = 1U << (sizeof(MEMBER_SET_TYPE) * 8 - 1);
+
+ if (set_id == RTE_MEMBER_NO_MATCH || (set_id & flag_mask) != 0)
+ return -EINVAL;
+
+ get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
+
+ /* if it is cache based filter, we try overwriting existing entry */
+ if (ss->cache) {
+ ret = try_overwrite(buckets, prim_bucket, sec_bucket, tmp_sig,
+ set_id);
+ if (ret != -1)
+ return ret;
+ }
+ /* If not full then insert into one slot*/
+ ret = try_insert(buckets, prim_bucket, sec_bucket, tmp_sig, set_id);
+ if (ret != -1)
+ return ret;
+
+ /* random pick prim or sec for recursive displacement */
+
+ uint32_t select_bucket = (tmp_sig && 1U) ? prim_bucket : sec_bucket;
+ if (ss->cache) {
+ ret = evict_from_bucket();
+ buckets[select_bucket].sigs[ret] = tmp_sig;
+ buckets[select_bucket].sets[ret] = set_id;
+ return 1;
+ }
+
+ ret = make_space_bucket(ss, select_bucket);
+ if (ret >= 0) {
+ buckets[select_bucket].sigs[ret] = tmp_sig;
+ buckets[select_bucket].sets[ret] = set_id;
+ ret = 1;
+ }
+
+ return ret;
+}
+
+
+void
+rte_member_free_ht(struct rte_member_setsum *ss)
+{
+ rte_free(ss->table);
+}
+
+
+int
+rte_member_delete_ht(struct rte_member_setsum *ss, const void *key,
+ MEMBER_SET_TYPE set_id)
+{
+ int i;
+ uint32_t prim_bucket, sec_bucket;
+ SIG_TYPE tmp_sig;
+ struct member_ht_bucket *buckets = ss->table;
+
+ get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
+
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ if (tmp_sig == buckets[prim_bucket].sigs[i] &&
+ set_id == buckets[prim_bucket].sets[i]) {
+ buckets[prim_bucket].sets[i] = RTE_MEMBER_NO_MATCH;
+ return 0;
+ }
+ }
+
+ for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+ if (tmp_sig == buckets[sec_bucket].sigs[i] &&
+ set_id == buckets[sec_bucket].sets[i]) {
+ buckets[sec_bucket].sets[i] = RTE_MEMBER_NO_MATCH;
+ return 0;
+ }
+ }
+ return -ENOENT;
+}
+
+
+void
+rte_member_reset_ht(const struct rte_member_setsum *ss)
+{
+ uint32_t i, j;
+ struct member_ht_bucket *buckets = ss->table;
+ for (i = 0; i < ss->bucket_cnt; i++) {
+ for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++)
+ buckets[i].sets[j] = RTE_MEMBER_NO_MATCH;
+ }
+}
diff --git a/lib/librte_member/rte_member_ht.h b/lib/librte_member/rte_member_ht.h
new file mode 100644
index 0000000..aebf1db
--- /dev/null
+++ b/lib/librte_member/rte_member_ht.h
@@ -0,0 +1,115 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _RTE_MEMBER_HT_H_
+#define _RTE_MEMBER_HT_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* Maximum number of pushes for cuckoo path in HT mode. */
+#define RTE_MEMBER_MAX_PUSHES 100
+
+typedef uint16_t SIG_TYPE; /* signature size is 16 bit */
+#define SIG_BITMASK 0xFFFF /* signature mask, 16 bit */
+
+
+/* The bucket struct for ht setsum */
+struct member_ht_bucket {
+ SIG_TYPE sigs[RTE_MEMBER_BUCKET_ENTRIES]; /* 2-byte signature */
+ MEMBER_SET_TYPE sets[RTE_MEMBER_BUCKET_ENTRIES]; /* 2-byte set */
+} __rte_cache_aligned;
+
+
+
+int
+rte_member_create_ht(struct rte_member_setsum *ss,
+ const struct rte_member_parameters *params);
+
+int
+rte_member_lookup_ht(const struct rte_member_setsum *setsum,
+ const void *key, MEMBER_SET_TYPE *set_id);
+
+
+
+uint32_t
+rte_member_lookup_bulk_ht(const struct rte_member_setsum *setsum,
+ const void **keys, uint32_t num_keys,
+ MEMBER_SET_TYPE *set_ids);
+
+
+
+
+uint32_t
+rte_member_lookup_multi_ht(const struct rte_member_setsum *setsum,
+ const void *key, uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id);
+
+
+
+uint32_t
+rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *setsum,
+ const void **keys, uint32_t num_keys, uint32_t match_per_key,
+ uint32_t *match_count,
+ MEMBER_SET_TYPE *set_ids);
+
+
+
+
+int
+rte_member_add_ht(const struct rte_member_setsum *setsum,
+ const void *key, MEMBER_SET_TYPE set_id);
+
+void
+rte_member_free_ht(struct rte_member_setsum *setsum);
+
+
+
+
+int
+rte_member_delete_ht(struct rte_member_setsum *ss, const void *key,
+ MEMBER_SET_TYPE set_id);
+
+
+void
+rte_member_reset_ht(const struct rte_member_setsum *setsum);
+
+
+#ifdef __cplusplus
+}
+#endif
+
+
+
+#endif /* _RTE_MEMBER_HT_H_ */
--
2.7.4
Yipeng Wang
2017-09-05 23:59:45 UTC
Permalink
Bloom Filter (BF) [1] is a well-known space-efficient
probabilistic data structure that answers set membership queries.
Vector of Bloom Filters (vBF) is an extension to traditional BF
that supports multi-set membership testing. Traditional BF will
return found or not-found for each key. vBF will also return
which set the key belongs to if it is found.

Since each set requires a BF, vBF should be used when set count
is small. vBF's false positive rate could be set arbitrarily so
that its memory requirement and lookup speed is better in certain
cases comparing to HT based set-summary.

This patch adds the vBF implementation.

[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable
Errors,” Communications of the ACM, 1970.

Signed-off-by: Yipeng Wang <***@intel.com>
---
lib/librte_member/Makefile | 2 +-
lib/librte_member/rte_member_vbf.c | 350 +++++++++++++++++++++++++++++++++++++
lib/librte_member/rte_member_vbf.h | 85 +++++++++
3 files changed, 436 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_member/rte_member_vbf.c
create mode 100644 lib/librte_member/rte_member_vbf.h

diff --git a/lib/librte_member/Makefile b/lib/librte_member/Makefile
index ad26548..50275ed 100644
--- a/lib/librte_member/Makefile
+++ b/lib/librte_member/Makefile
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map
LIBABIVER := 1

# all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c rte_member_vbf.c
# install includes
SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h

diff --git a/lib/librte_member/rte_member_vbf.c b/lib/librte_member/rte_member_vbf.c
new file mode 100644
index 0000000..49ca69e
--- /dev/null
+++ b/lib/librte_member/rte_member_vbf.c
@@ -0,0 +1,350 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <math.h>
+#include <string.h>
+
+#include <rte_malloc.h>
+#include <rte_memory.h>
+#include <rte_errno.h>
+#include <rte_log.h>
+
+#include "rte_member.h"
+#include "rte_member_vbf.h"
+
+/*
+ * vBF currently implemented as a big array.
+ * The BFs have a vertical layout. Bits in same location of all bfs will stay
+ * in the same cache line.
+ * For example, if we have 32 bloom filters, we use a uint32_t array to
+ * represent all of them. array[0] represent the first location of all the
+ * bloom filters, array[1] represents the second location of all the
+ * bloom filters, etc. The advantage of this layout is to minimize the average
+ * number of memory accesses to test all bloom filters.
+ *
+ * Currently the implementation supports vBF containing 1,2,4,8,16,32 BFs.
+ */
+
+int
+rte_member_create_vbf(struct rte_member_setsum *ss,
+ const struct rte_member_parameters *params)
+{
+
+ if (params->num_set > 32 || !rte_is_power_of_2(params->num_set) ||
+ params->false_positive_rate == 0 ||
+ params->false_positive_rate > 1) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, MEMBER, "vBF create with invalid parameters\n");
+ return -EINVAL;
+ }
+
+ /* We assume expected keys evenly distribute to all BFs */
+ uint32_t num_keys_per_bf = (params->num_keys + ss->num_set - 1) /
+ ss->num_set;
+
+ if (num_keys_per_bf == 0) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, MEMBER, "BF expected number of key is zero\n");
+ return -EINVAL;
+ }
+ /*
+ * Note that the false positive rate is for all BFs in the vBF
+ * such that the single BF's false positive rate need to be
+ * calculated.
+ * Assume each BF's False positive rate is x. The total false positive
+ * rate is fp = 1-(1-x)^n.
+ * => x = 1 - (1-fp)^(1/n)
+ */
+
+ float x = 1 - pow((1 - params->false_positive_rate), 1.0 / ss->num_set);
+
+ if (x == 0) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, MEMBER, "BF false positive rate is zero\n");
+ return -EINVAL;
+ }
+
+ uint32_t bits = ceil((num_keys_per_bf *
+ log(x)) / log(1.0 / (pow(2.0, log(2.0)))));
+
+ /* We round to power of 2 for performance during lookup */
+ ss->bits = rte_align32pow2(bits);
+ if (ss->bits == 0) {
+ rte_errno = EINVAL;
+ RTE_LOG(ERR, MEMBER, "BF size is zero\n");
+ return -EINVAL;
+ }
+
+ ss->num_hashes = (uint32_t)(log(2.0) * bits / num_keys_per_bf);
+ ss->bit_mask = ss->bits - 1;
+
+
+ /*
+ * Since we round the bits to power of 2, the final false positive
+ * rate will probably not be same as the user specified. We log the
+ * new value as debug message.
+ */
+ float new_fp = pow((1 - pow((1 - 1.0 / ss->bits), num_keys_per_bf *
+ ss->num_hashes)), ss->num_hashes);
+ new_fp = 1 - pow((1 - new_fp), ss->num_set);
+
+ /*
+ * reduce hash function count, until we approach the user specified
+ * false-positive rate. otherwise it is too conservative
+ */
+ int tmp_num_hash = ss->num_hashes;
+
+ while (tmp_num_hash > 1) {
+ float tmp_fp = new_fp;
+
+ tmp_num_hash--;
+ new_fp = pow((1 - pow((1 - 1.0 / ss->bits), num_keys_per_bf *
+ tmp_num_hash)), tmp_num_hash);
+ new_fp = 1 - pow((1 - new_fp), ss->num_set);
+
+ if (new_fp > params->false_positive_rate) {
+ new_fp = tmp_fp;
+ tmp_num_hash++;
+ break;
+ }
+ }
+
+ ss->num_hashes = tmp_num_hash;
+
+ RTE_LOG(DEBUG, MEMBER, "vector bloom filter created, "
+ "each bloom filter expects %u keys, needs %u bits, %u hashes, "
+ "with false positive rate set as %.5f, "
+ "The new calculated vBF false positive rate is %.5f\n",
+ num_keys_per_bf, ss->bits, ss->num_hashes, x, new_fp);
+
+ ss->table = rte_zmalloc_socket(NULL, sizeof(uint32_t) * ss->num_set *
+ (ss->bits >> 5), RTE_CACHE_LINE_SIZE, ss->socket_id);
+
+ if (ss->table == NULL)
+ return -ENOMEM;
+
+ return 0;
+}
+
+
+/*
+ * a is how many bits could be represented by one uint32_t variable.
+ * b is used for division shift
+ * shift is used for multiplication shift
+ */
+static inline uint32_t
+test_bit(uint32_t h1, uint32_t h2, uint32_t iter, uint32_t shift, uint32_t a,
+ uint32_t b, const struct rte_member_setsum *ss)
+{
+ uint32_t *vbf = ss->table;
+ uint32_t n = ss->num_set;
+ uint32_t bit_loc = (h1 + iter * h2) & ss->bit_mask;
+ /*
+ * x>>b is the divide, x & (a-1) is the mod, & (1<<n-1) to mask out bits
+ * we do not need
+ */
+ return (vbf[bit_loc>>b] >> ((bit_loc & (a - 1)) << shift)) &
+ ((1ULL << n) - 1);
+}
+
+
+static inline void
+set_bit(uint32_t h1, uint32_t h2, uint32_t iter, uint32_t shift, uint32_t a,
+ uint32_t b, const struct rte_member_setsum *ss, int32_t set)
+{
+ uint32_t *vbf = ss->table;
+ uint32_t bit_loc = (h1 + iter * h2) & ss->bit_mask;
+ vbf[bit_loc>>b] |= 1U << (((bit_loc & (a - 1)) << shift) + set - 1);
+}
+
+
+
+int
+rte_member_lookup_vbf(const struct rte_member_setsum *ss, const void *key,
+ MEMBER_SET_TYPE *set_id)
+{
+ uint32_t j;
+ uint32_t h1 = MEMBER_PRIM_HASH(key, ss->key_len, ss->prim_hash_seed);
+ uint32_t h2 = MEMBER_SEC_HASH(&h1, 4, ss->sec_hash_seed);
+ uint32_t mask = 0xFFFFFFFF;
+ uint32_t shift = __builtin_ctz(ss->num_set);
+ uint32_t a = 32 >> shift;
+ uint32_t b = __builtin_ctz(a);
+
+ for (j = 0; j < ss->num_hashes; j++)
+ mask &= test_bit(h1, h2, j, shift, a, b, ss);
+
+ if (mask) {
+ *set_id = __builtin_ctz(mask) + 1;
+ return 1;
+ }
+
+ *set_id = RTE_MEMBER_NO_MATCH;
+ return 0;
+}
+
+
+
+uint32_t
+rte_member_lookup_bulk_vbf(const struct rte_member_setsum *ss,
+ const void **keys, uint32_t num_keys, MEMBER_SET_TYPE *set_ids)
+{
+ uint32_t i, k, h1, h2;
+ uint32_t ret = 0;
+ uint32_t mask;
+ uint32_t shift = __builtin_ctz(ss->num_set);
+ uint32_t a = 32 >> shift;
+ uint32_t b = __builtin_ctz(a);
+
+ for (i = 0; i < num_keys; i++) {
+ mask = 0xFFFFFFFF;
+ h1 = MEMBER_PRIM_HASH(keys[i], ss->key_len,
+ ss->prim_hash_seed);
+ h2 = MEMBER_SEC_HASH(&h1, 4, ss->sec_hash_seed);
+ for (k = 0; k < ss->num_hashes; k++)
+ mask &= test_bit(h1, h2, k, shift, a, b, ss);
+
+ if (mask) {
+ set_ids[i] = __builtin_ctz(mask) + 1;
+ ret++;
+ } else
+ set_ids[i] = RTE_MEMBER_NO_MATCH;
+ }
+ return ret;
+}
+
+
+
+uint32_t
+rte_member_lookup_multi_vbf(const struct rte_member_setsum *ss,
+ const void *key, uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id)
+{
+ uint32_t ret = 0;
+ uint32_t j;
+ uint32_t h1 = MEMBER_PRIM_HASH(key, ss->key_len, ss->prim_hash_seed);
+ uint32_t h2 = MEMBER_SEC_HASH(&h1, 4, ss->sec_hash_seed);
+ uint32_t mask = 0xFFFFFFFF;
+ uint32_t shift = __builtin_ctz(ss->num_set);
+ uint32_t a = 32 >> shift;
+ uint32_t b = __builtin_ctz(a);
+
+ for (j = 0; j < ss->num_hashes; j++)
+ mask &= test_bit(h1, h2, j, shift, a, b, ss);
+
+ while (mask) {
+ uint32_t loc = __builtin_ctz(mask);
+ set_id[ret] = loc + 1;
+ ret++;
+ if (ret >= match_per_key)
+ return ret;
+ mask &= ~(1U << loc);
+ }
+ return ret;
+}
+
+
+
+uint32_t
+rte_member_lookup_multi_bulk_vbf(const struct rte_member_setsum *ss,
+ const void **keys, uint32_t num_keys, uint32_t match_per_key,
+ uint32_t *match_count,
+ MEMBER_SET_TYPE *set_ids)
+{
+ uint32_t i, k, h1, h2;
+ uint32_t ret = 0;
+ uint32_t match_cnt_t;
+ uint32_t mask;
+ uint32_t shift = __builtin_ctz(ss->num_set);
+ uint32_t a = 32 >> shift;
+ uint32_t b = __builtin_ctz(a);
+
+ for (i = 0; i < num_keys; i++) {
+ match_cnt_t = 0;
+ mask = 0xFFFFFFFF;
+ h1 = MEMBER_PRIM_HASH(keys[i], ss->key_len,
+ ss->prim_hash_seed);
+ h2 = MEMBER_SEC_HASH(&h1, 4, ss->sec_hash_seed);
+
+ for (k = 0; k < ss->num_hashes; k++)
+ mask &= test_bit(h1, h2, k, shift, a, b, ss);
+
+ while (mask) {
+ uint32_t loc = __builtin_ctz(mask);
+ if (match_cnt_t >= match_per_key)
+ break;
+ set_ids[i * match_per_key + match_cnt_t] = loc + 1;
+ match_cnt_t++;
+ mask &= ~(1U << loc);
+ }
+ match_count[i] = match_cnt_t;
+ if (match_cnt_t != 0)
+ ret++;
+ }
+ return ret;
+}
+
+
+int
+rte_member_add_vbf(const struct rte_member_setsum *ss,
+ const void *key, MEMBER_SET_TYPE set_id)
+{
+ uint32_t i, h1, h2;
+
+ if (set_id > ss->num_set || set_id == RTE_MEMBER_NO_MATCH)
+ return -EINVAL;
+
+ h1 = MEMBER_PRIM_HASH(key, ss->key_len, ss->prim_hash_seed);
+ h2 = MEMBER_SEC_HASH(&h1, 4, ss->sec_hash_seed);
+ uint32_t shift = __builtin_ctz(ss->num_set);
+ uint32_t a = 32 >> shift;
+ uint32_t b = __builtin_ctz(a);
+
+ for (i = 0; i < ss->num_hashes; i++)
+ set_bit(h1, h2, i, shift, a, b, ss, set_id);
+ return 0;
+}
+
+void
+rte_member_free_vbf(struct rte_member_setsum *ss)
+{
+ rte_free(ss->table);
+}
+
+
+void
+rte_member_reset_vbf(const struct rte_member_setsum *ss)
+{
+ uint32_t *vbf = ss->table;
+ memset(vbf, 0, (ss->num_set * ss->bits) >> 3);
+}
diff --git a/lib/librte_member/rte_member_vbf.h b/lib/librte_member/rte_member_vbf.h
new file mode 100644
index 0000000..3d3ee33
--- /dev/null
+++ b/lib/librte_member/rte_member_vbf.h
@@ -0,0 +1,85 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _RTE_MEMBER_VBF_H_
+#define _RTE_MEMBER_VBF_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+int
+rte_member_create_vbf(struct rte_member_setsum *ss,
+ const struct rte_member_parameters *params);
+
+
+int
+rte_member_lookup_vbf(const struct rte_member_setsum *setsum,
+ const void *key, MEMBER_SET_TYPE *set_id);
+
+
+uint32_t
+rte_member_lookup_bulk_vbf(const struct rte_member_setsum *setsum,
+ const void **keys, uint32_t num_keys,
+ MEMBER_SET_TYPE *set_ids);
+
+
+uint32_t
+rte_member_lookup_multi_vbf(const struct rte_member_setsum *setsum,
+ const void *key, uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id);
+
+
+uint32_t
+rte_member_lookup_multi_bulk_vbf(const struct rte_member_setsum *setsum,
+ const void **keys, uint32_t num_keys, uint32_t match_per_key,
+ uint32_t *match_count,
+ MEMBER_SET_TYPE *set_ids);
+
+int
+rte_member_add_vbf(const struct rte_member_setsum *setsum,
+ const void *key, MEMBER_SET_TYPE set_id);
+
+void
+rte_member_free_vbf(struct rte_member_setsum *ss);
+
+
+void
+rte_member_reset_vbf(const struct rte_member_setsum *setsum);
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_MEMBER_VBF_H_ */
--
2.7.4
Yipeng Wang
2017-09-05 23:59:46 UTC
Permalink
For key search, the signatures of all entries are compared against
the signature of the key that is being looked up. Since all
signatures are contguously put in a bucket, they can be compared
with vector instructions (AVX2), achieving higher lookup performance.

This patch adds AVX2 implementation in a separate header file.

Signed-off-by: Yipeng Wang <***@intel.com>
---
lib/librte_member/rte_member_ht.c | 143 ++++++++++++++++++++++++++++---------
lib/librte_member/rte_member_x86.h | 111 ++++++++++++++++++++++++++++
2 files changed, 222 insertions(+), 32 deletions(-)
create mode 100644 lib/librte_member/rte_member_x86.h

diff --git a/lib/librte_member/rte_member_ht.c b/lib/librte_member/rte_member_ht.c
index b2ae6d0..15e2534 100644
--- a/lib/librte_member/rte_member_ht.c
+++ b/lib/librte_member/rte_member_ht.c
@@ -40,6 +40,10 @@
#include "rte_member.h"
#include "rte_member_ht.h"

+#if defined(RTE_ARCH_X86)
+#include "rte_member_x86.h"
+#endif
+

static inline int
insert_overwrite_search(uint32_t bucket, SIG_TYPE tmp_sig,
@@ -135,6 +139,13 @@ rte_member_create_ht(struct rte_member_setsum *ss,
for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++)
buckets[i].sets[j] = RTE_MEMBER_NO_MATCH;
}
+#if defined(RTE_ARCH_X86)
+ if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2) &&
+ RTE_MEMBER_BUCKET_ENTRIES == 16)
+ ss->sig_cmp_fn = RTE_MEMBER_COMPARE_AVX2;
+ else
+#endif
+ ss->sig_cmp_fn = RTE_MEMBER_COMPARE_SCALAR;


RTE_LOG(DEBUG, MEMBER, "Hash table based filter created, "
@@ -174,11 +185,23 @@ rte_member_lookup_ht(const struct rte_member_setsum *ss,
*set_id = RTE_MEMBER_NO_MATCH;
get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);

- if (search_bucket_single(prim_bucket, tmp_sig, buckets,
- set_id) ||
- search_bucket_single(sec_bucket, tmp_sig,
- buckets, set_id))
- return 1;
+ switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+ case RTE_MEMBER_COMPARE_AVX2:
+ if (search_bucket_single_avx(prim_bucket, tmp_sig, buckets,
+ set_id) ||
+ search_bucket_single_avx(sec_bucket, tmp_sig,
+ buckets, set_id))
+ return 1;
+ break;
+#endif
+ default:
+ if (search_bucket_single(prim_bucket, tmp_sig, buckets,
+ set_id) ||
+ search_bucket_single(sec_bucket, tmp_sig,
+ buckets, set_id))
+ return 1;
+ }

return 0;
}
@@ -203,13 +226,27 @@ rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss,
}

for (i = 0; i < num_keys; i++) {
- if (search_bucket_single(prim_buckets[i], tmp_sig[i],
- buckets, &set_id[i]) ||
- search_bucket_single(sec_buckets[i],
- tmp_sig[i], buckets, &set_id[i]))
- ret++;
- else
- set_id[i] = RTE_MEMBER_NO_MATCH;
+ switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+ case RTE_MEMBER_COMPARE_AVX2:
+ if (search_bucket_single_avx(prim_buckets[i],
+ tmp_sig[i], buckets, &set_id[i]) ||
+ search_bucket_single_avx(sec_buckets[i],
+ tmp_sig[i], buckets, &set_id[i]))
+ ret++;
+ else
+ set_id[i] = RTE_MEMBER_NO_MATCH;
+ break;
+#endif
+ default:
+ if (search_bucket_single(prim_buckets[i], tmp_sig[i],
+ buckets, &set_id[i]) ||
+ search_bucket_single(sec_buckets[i],
+ tmp_sig[i], buckets, &set_id[i]))
+ ret++;
+ else
+ set_id[i] = RTE_MEMBER_NO_MATCH;
+ }
}
return ret;
}
@@ -227,12 +264,24 @@ rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,

get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);

- search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
- match_per_key, set_id);
- if (ret < match_per_key)
- search_bucket_multi(sec_bucket, tmp_sig,
- buckets, &ret, match_per_key, set_id);
- return ret;
+ switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+ case RTE_MEMBER_COMPARE_AVX2:
+ search_bucket_multi_avx(prim_bucket, tmp_sig, buckets,
+ &ret, match_per_key, set_id);
+ if (ret < match_per_key)
+ search_bucket_multi_avx(sec_bucket, tmp_sig,
+ buckets, &ret, match_per_key, set_id);
+ return ret;
+#endif
+ default:
+ search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
+ match_per_key, set_id);
+ if (ret < match_per_key)
+ search_bucket_multi(sec_bucket, tmp_sig,
+ buckets, &ret, match_per_key, set_id);
+ return ret;
+ }
}


@@ -259,16 +308,34 @@ rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,
for (i = 0; i < num_keys; i++) {
match_cnt_t = 0;

- search_bucket_multi(prim_buckets[i], tmp_sig[i],
- buckets, &match_cnt_t, match_per_key,
- &set_ids[i*match_per_key]);
- if (match_cnt_t < match_per_key)
- search_bucket_multi(sec_buckets[i], tmp_sig[i],
+ switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+ case RTE_MEMBER_COMPARE_AVX2:
+ search_bucket_multi_avx(prim_buckets[i], tmp_sig[i],
buckets, &match_cnt_t, match_per_key,
&set_ids[i*match_per_key]);
- match_count[i] = match_cnt_t;
- if (match_cnt_t != 0)
- ret++;
+ if (match_cnt_t < match_per_key)
+ search_bucket_multi_avx(sec_buckets[i],
+ tmp_sig[i], buckets, &match_cnt_t,
+ match_per_key,
+ &set_ids[i*match_per_key]);
+ match_count[i] = match_cnt_t;
+ if (match_cnt_t != 0)
+ ret++;
+ break;
+#endif
+ default:
+ search_bucket_multi(prim_buckets[i], tmp_sig[i],
+ buckets, &match_cnt_t, match_per_key,
+ &set_ids[i*match_per_key]);
+ if (match_cnt_t < match_per_key)
+ search_bucket_multi(sec_buckets[i], tmp_sig[i],
+ buckets, &match_cnt_t, match_per_key,
+ &set_ids[i*match_per_key]);
+ match_count[i] = match_cnt_t;
+ if (match_cnt_t != 0)
+ ret++;
+ }
}
return ret;
}
@@ -300,12 +367,24 @@ try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,

static inline int
try_overwrite(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
- SIG_TYPE sig, MEMBER_SET_TYPE set_id)
+ SIG_TYPE sig, MEMBER_SET_TYPE set_id,
+ enum rte_member_sig_compare_function cmp_fn)
{
- if (insert_overwrite_search(prim, sig, buckets, set_id) ||
- insert_overwrite_search(sec, sig, buckets,
- set_id))
- return 0;
+ switch (cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+ case RTE_MEMBER_COMPARE_AVX2:
+ if (insert_overwrite_search_avx(prim, sig, buckets, set_id) ||
+ insert_overwrite_search_avx(sec, sig, buckets,
+ set_id))
+ return 0;
+ break;
+#endif
+ default:
+ if (insert_overwrite_search(prim, sig, buckets, set_id) ||
+ insert_overwrite_search(sec, sig, buckets,
+ set_id))
+ return 0;
+ }
return -1;
}

@@ -411,7 +490,7 @@ rte_member_add_ht(const struct rte_member_setsum *ss,
/* if it is cache based filter, we try overwriting existing entry */
if (ss->cache) {
ret = try_overwrite(buckets, prim_bucket, sec_bucket, tmp_sig,
- set_id);
+ set_id, ss->sig_cmp_fn);
if (ret != -1)
return ret;
}
diff --git a/lib/librte_member/rte_member_x86.h b/lib/librte_member/rte_member_x86.h
new file mode 100644
index 0000000..c55f128
--- /dev/null
+++ b/lib/librte_member/rte_member_x86.h
@@ -0,0 +1,111 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _RTE_MEMBER_X86_H_
+#define _RTE_MEMBER_X86_H_
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <x86intrin.h>
+
+
+#if defined(RTE_MACHINE_CPUFLAG_AVX2)
+
+
+static inline int
+insert_overwrite_search_avx(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ MEMBER_SET_TYPE set_id)
+{
+ uint32_t hitmask = _mm256_movemask_epi8((__m256i)_mm256_cmpeq_epi16(
+ _mm256_load_si256((__m256i const *)buckets[bucket].sigs),
+ _mm256_set1_epi16(tmp_sig)));
+ if (hitmask) {
+ uint32_t hit_idx = __builtin_ctzl(hitmask) / 2;
+ buckets[bucket].sets[hit_idx] = set_id;
+ return 1;
+ }
+ return 0;
+}
+
+
+static inline int
+search_bucket_single_avx(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ MEMBER_SET_TYPE *set_id)
+{
+ uint32_t hitmask = _mm256_movemask_epi8((__m256i)_mm256_cmpeq_epi16(
+ _mm256_load_si256((__m256i const *)buckets[bucket].sigs),
+ _mm256_set1_epi16(tmp_sig)));
+ while (hitmask) {
+ uint32_t hit_idx = __builtin_ctzl(hitmask) / 2;
+ if (buckets[bucket].sets[hit_idx] != RTE_MEMBER_NO_MATCH) {
+ *set_id = buckets[bucket].sets[hit_idx];
+ return 1;
+ }
+ hitmask &= ~(3U << (hit_idx) * 2);
+ }
+ return 0;
+}
+
+static inline void
+search_bucket_multi_avx(uint32_t bucket, SIG_TYPE tmp_sig,
+ struct member_ht_bucket *buckets,
+ uint32_t *counter,
+ uint32_t match_per_key,
+ MEMBER_SET_TYPE *set_id)
+{
+ uint32_t hitmask = _mm256_movemask_epi8((__m256i)_mm256_cmpeq_epi16(
+ _mm256_load_si256((__m256i const *)buckets[bucket].sigs),
+ _mm256_set1_epi16(tmp_sig)));
+ while (hitmask) {
+ uint32_t hit_idx = __builtin_ctzl(hitmask) / 2;
+ if (buckets[bucket].sets[hit_idx] != RTE_MEMBER_NO_MATCH) {
+ set_id[*counter] = buckets[bucket].sets[hit_idx];
+ (*counter)++;
+ if (*counter >= match_per_key)
+ return;
+ }
+ hitmask &= ~(3U << (hit_idx) * 2);
+ }
+}
+#endif
+
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_MEMBER_X86_H_ */
--
2.7.4
Yipeng Wang
2017-09-05 23:59:47 UTC
Permalink
This patch enables the Membership library.

Signed-off-by: Yipeng Wang <***@intel.com>
---
MAINTAINERS | 7 +++++++
config/common_base | 5 +++++
lib/librte_member/Makefile | 2 ++
mk/rte.app.mk | 2 ++
4 files changed, 16 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index a0cd75e..e372edf 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -734,6 +734,13 @@ F: test/test/test_meter.c
F: examples/qos_meter/
F: doc/guides/sample_app_ug/qos_metering.rst

+Membership - EXPERIMENTAL
+M: Yipeng Wang <***@intel.com>
+M: Sameh Gobriel <***@intel.com>
+F: lib/librte_member/
+F: doc/guides/prog_guide/member_lib.rst
+F: test/test/test_member*
+

Other libraries
---------------
diff --git a/config/common_base b/config/common_base
index 5e97a08..5e31ced 100644
--- a/config/common_base
+++ b/config/common_base
@@ -595,6 +595,11 @@ CONFIG_RTE_LIBRTE_HASH_DEBUG=n
CONFIG_RTE_LIBRTE_EFD=y

#
+# Compile librte_member
+#
+CONFIG_RTE_LIBRTE_MEMBER=y
+
+#
# Compile librte_jobstats
#
CONFIG_RTE_LIBRTE_JOBSTATS=y
diff --git a/lib/librte_member/Makefile b/lib/librte_member/Makefile
index 50275ed..3bac1d0 100644
--- a/lib/librte_member/Makefile
+++ b/lib/librte_member/Makefile
@@ -37,6 +37,8 @@ LIB = librte_member.a
CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3

+LDLIBS += -lm
+
EXPORT_MAP := rte_member_version.map

LIBABIVER := 1
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index c25fdd9..c79acf0 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -86,6 +86,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CFGFILE) += -lrte_cfgfile
_LDLIBS-y += --whole-archive

_LDLIBS-$(CONFIG_RTE_LIBRTE_HASH) += -lrte_hash
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lrte_member
_LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lrte_vhost
_LDLIBS-$(CONFIG_RTE_LIBRTE_KVARGS) += -lrte_kvargs
_LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF) += -lrte_mbuf
@@ -196,6 +197,7 @@ endif
_LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lm
_LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrt
_LDLIBS-$(CONFIG_RTE_LIBRTE_METER) += -lm
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lm
ifeq ($(CONFIG_RTE_LIBRTE_VHOST_NUMA),y)
_LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lnuma
endif
--
2.7.4
Thomas Monjalon
2017-09-22 10:48:56 UTC
Permalink
Post by Yipeng Wang
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -734,6 +734,13 @@ F: test/test/test_meter.c
F: examples/qos_meter/
F: doc/guides/sample_app_ug/qos_metering.rst
+Membership - EXPERIMENTAL
+F: lib/librte_member/
+F: doc/guides/prog_guide/member_lib.rst
+F: test/test/test_member*
I know it is not obvious but this section is sorted in alphabetical order.
Please move between Hashes and LPM. Thanks
Yipeng Wang
2017-09-05 23:59:48 UTC
Permalink
This patch adds functional and performance tests for membership
library.

Signed-off-by: Yipeng Wang <***@intel.com>
---
test/test/Makefile | 3 +
test/test/test_member.c | 682 +++++++++++++++++++++++++++++++++++++++++++
test/test/test_member_perf.c | 643 ++++++++++++++++++++++++++++++++++++++++
3 files changed, 1328 insertions(+)
create mode 100644 test/test/test_member.c
create mode 100644 test/test/test_member_perf.c

diff --git a/test/test/Makefile b/test/test/Makefile
index 42d9a49..b61dde3 100644
--- a/test/test/Makefile
+++ b/test/test/Makefile
@@ -123,6 +123,9 @@ SRCS-y += test_logs.c
SRCS-y += test_memcpy.c
SRCS-y += test_memcpy_perf.c

+
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member_perf.c
SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd.c
SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd_perf.c

diff --git a/test/test/test_member.c b/test/test/test_member.c
new file mode 100644
index 0000000..8965e81
--- /dev/null
+++ b/test/test/test_member.c
@@ -0,0 +1,682 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+/* This test is for membership library's simple feature test */
+
+#include <rte_memcpy.h>
+#include <rte_malloc.h>
+#include <rte_member.h>
+#include <rte_byteorder.h>
+#include <rte_random.h>
+#include <rte_debug.h>
+#include <rte_ip.h>
+
+#include "test.h"
+
+void *setsum_ht;
+void *setsum_cache;
+void *setsum_vbf;
+
+/* 5-tuple key type */
+struct flow_key {
+ uint32_t ip_src;
+ uint32_t ip_dst;
+ uint16_t port_src;
+ uint16_t port_dst;
+ uint8_t p