UA Part 14: PubSub Main Technology Features
In general speaking the following two distinct patterns are used to transfer data between communicating parties:
connection-oriented: requires a session that has to be established before any data can be sent between sender and receiver
connectionless-oriented: the sender may start sending messages (called packets or datagrams) to the destination without any preceding handshake procedure
Each has its own advantages and disadvantages. In general, the OPC UA is a session centric communication. The session is established by the OPC UA Client that must connect to the OPC UA Server before any data can be exchanged between them. In this Client/Server scenario defined by the Services in Part 4, the data flow is bidirectional over the session. The session entities communicate over a secure channel that is created in the underlying communication layer and relies upon it for secure communication. It enables to log-in using user authentication and operations authorization. More details you can find in the section:
Using the connection-oriented communication pattern it is difficult or even impossible to gather and process data from mobile things (e.g. smart devices, cigarettes box, drug blister, etc.), which is one of the Internet of Things paradigms. More details you can find in IoT versus SCADA/DCS Data Acquisition Patterns.
The OPC UA PubSub specification offers the connectionless approach as an additional option to session based client-server interoperability and claims that it is a consistent part of the OPC UA specifications suit. As a result, it may be recognized as the IoT ready technology.
OPC UA Part 14: PubSub promotes interoperability of loosely coupled PubSub Applications. By design, they often will not even know each other. In this case, it is impossible to create in-band interoperability alliance based on the direct negotiations of required configuration parameters and security artifacts. Their primary relationship is the shared understanding of:
specific semantics (meaning) of exchanged data
the syntax and semantics of messages that include these data
common underlying messages transport layer
The specification claims that the PubSub integrates into the existing OPC UA technology but as result of applying the connectionless communication it is easier to implement low power and low-latency communications on local networks. Additionally, the specification states that PubSub is based on the OPC UA Information Model with the aim of seamless integration into OPC UA Servers and OPC UA Clients. Nevertheless, the PubSub communication does not require such a role dependency, i.e. there is no necessity for Publisher or Subscriber to be either an OPC UA Server or an OPC UA Client to participate in the communication.
Note 1: Unfortunately, OPC UA Information Model is not used to promote PubSub Applications in-band interoperability. This concept is only employed to define
Security Key ManagementandConfiguration Managementmodels, which have an only indirect impact on the in-band PubSub Applications interoperability.
PubSub Applications exchange messages formatted as the NetworkMessage structure using underlying communication stack. As illustrated in the following domain model (Figure 1), directly or indirectly the specification defines the following actors:
Publisher: pushes the current process data formatted as theNetworkMessagestructure to an underlying communication stackSubscriber: consumes the process data, which is recovered from theNetworkMessagestructures polled from the underlying communication stackDistribution Channel- selected common underlying communication stackNetworkMessage- a data structure formatted in compliance with the syntax and semantics defined by the OPC.UA.PubSub specificationSecurity Key Management- a service that provides security keys used to sign and encryptNetworkMessagedata structuresConfiguration Management- an external application used to remotely configure PubSub Application
The Publisher is the actor that pushes NetworkMessage structures to an underlying communication stack responsible to transport it over the network. It represents a certain data source, for example, a control device, a manufacturing process, a weather station or a stock exchange. It may be also OPC UA Client, OPC UA Server or in general any application that understand the syntax and semantics of the NetworkMessage structure.
The Subscriber actors are the consumers of NetworkMessage structures, which are polled from the underlying transport layer. They may be OPC UA Client, OPC UA Server or in general any applications that understand the syntax and semantics of the NetworkMessage structure.
To interchange the process data Publisher and all associated Subscribers nodes depend on a common Distribution Channel. Distribution Channel models common knowledge necessary to use an underlying messages transport communication stack, i.e. underlying protocol stack and relevant parameters to route the messages over the network.
A Security Key Management provides keys for message security that can be used by the Publisher to sign and encrypt NetworkMessage structures and by the Subscriber to verify the signature of and decrypt the NetworkMessage. NetworkMessage security concerns the integrity and confidentiality of the published message payload. The level of security can be:
no security
signing but no encryption
signing and encryption
Message security is end-to-end security (from Publisher to Subscriber instances) and requires common knowledge of the cryptographic artifacts necessary to sign and encrypt on the Publisher side as well as validate the signature and decrypt on the Subscriber side. The message security is independent of the transport protocol mapping and is defined by the specification.
The specification defines OPC UA Information Model for Security Key Management services and many possible scenarios that can be used to select the security profile and provide appropriate security artifacts to the Publisher and Subscriber using this model. One of them is to implement this model as the OPC UA Server or OPC UA Client where the OPC UA IM model is used to describe the server OPC UA Address Space. A detailed description of all possible scenarios applicable to select security profile and exchange security artifacts is outside of the scope of this section.
Publisher and Subscriber nodes may be configurable through vendor-specific engineering tools or using the dedicated configuration OPC UA Information Model described in this standard. This model allows a standard OPC UA Client based configuration tool to configure a PubSub Application connecting to the embedded OPC UA Server. Using remote Configuration Tool over an OPC UA Session does not determine how dynamic the configuration can be. More detailed description of this model is outside of the scope of this section.
It is worth stressing that the configuration model doesn't provide any definition dedicated to being used for the process data bindings configuration.
The PubSub Applications are decoupled by exchanging messages over a selected underlying protocol stack. It is worth stressing that by design the PubSub Application doesn't expose any API that can be used to transfer upper layer data over the network, i.e. it is not a communication layer in the communication stack. It means that these applications must produce and/or consume the process data, i.e. the NetworkMessage must be populated using external process data.
PubSub Applications interoperability doesn't depend on any functionality provided by the underlying transport layer. According to the specification, the Subscriber and Publisher can be interconnected using any transparent messages transport infrastructure. The specification defines two groups of solutions:
In both cases, the one-to-many relationship between Publisher and Subscriber can be obtained. For UDP multicast messages distribution may be applied to send Internet Protocol (IP) (figure below) datagrams to a group of interested receivers in a single transmission. For the broker-based transport, all messages are published to specific queues (e.g. topics, nodes) that the broker exposes and Subscribers can listen to these queues.
The OPC UA PubSub specification lists the following protocol stacks that can be selected as the transport for messages and their possible combinations with message mappings:
OPC UA UDP - simple UDP based protocol that is used to transport UADP
NetworkMessagesOPC UA Ethernet - simple Ethernet based protocol using EtherType B62C that is used to transport UADP
NetworkMessagesas payload of the Ethernet II frame without IP or UDP headersAMQP - Advanced Message Queuing Protocol (AMQP) based protocol that is used to transport JSON and UADP
NetworkMessagestructuresMQTT - Message Queue Telemetry Transport (MQTT) based protocol that is used to transport JSON and UADP
NetworkMessagestructures
Because the specification doesn't define normative references for OPC UA UDP and OPC UA Ethernet in section References they are inferred from the context. Based on this mapping in the figure below the architecture of protocol stack is determined as the domain diagram. The diagram has been worked out on the best effort approach.
Note 2 - for the sake of simplicity the diagram contains only protocols relevant for the mapping in concern. In other words, the classes representing the abstract OSI model layers (
OSI Transport,OSI Network, andOSI Data Link) may aggregate and use a variety of protocols depending on the local network infrastructure, e.g. IEEE 802.11 forOSI Data Link Layer.
Following the specification, the transport protocol mapping is modeled as the four top-level classes called appropriately Ethernet, UDP, MQTT, AMQP. They may be recognized as the underlying API of the protocol stack and are aggregated into one common communication layer used to exchange the messages over the network (section Semantic-Data Message Centric Communication ).
Here it must be stressed that the mentioned in the section title term transport protocol has nothing in common with the Open System Interconnection Reference Model (OSI model) Transport Layer. Referring to the OSI model the MQTT and AMQP protocols should be recognized as the OSI Application Layer protocols. The OSI Application Layer is the one at the top of the model. For the sake of simplicity, the OSI Application Layer is not present in the diagram. Because the PubSub specification defines also the protocol on the same layer some functionality is redundant in this case - they overlap on each other. For the purpose of traversing the network by the messages, the PubSub Application uses MQTT and AMQP protocols as a transparent communication service. Applying the broker-based approach also means that some functionality related to communication reliability, data selection, and distribution is delegated to them. Details related to MQTT mapping are covered by the section Underlying Transport over MQTT. Details related to AMQP mapping are covered by the section Underlying Transport over AMQP.
The OSI Presentation Layer represents the services that are responsible for the translation of the application data encoding to network encoding, and translation back from the network encoding to application encoding. In other words, the layer “presents” data for the application or the network. This functionality (encoding/decoding) is embedded in the definition of the PubSub message syntax rules. This syntax rules (OPC UA Part 6) are common for all the OPC UA specifications suite. For the sake of simplicity, the OSI Presentation Layer is not present in the diagram.
In the published/subscriber communication pattern the OSI Session Layer is empty, so it is ignored in the domain model presented in figure above.
The specification doesn't define particular mapping rules referring to protocol stack used by the AMQP and MQTT, so an abstract OSI Transport Layer is used in the proposed model as the underlying communication layer for them. In this case, all requirements against relevant specifications apply.
On the other hand, according to the mapping rules the User Datagram Protocol (UDP) protocol (UDP) is pointed out by the PubSub specification as the only concrete implementation of the OSI Transport Layer. In this case, the protocol can be recognized as the base for the UDP mapping rules stated by the specification and a not sharable part of the abstract OSI Transport Layer.
The specification doesn't define any subscription management services, namely, it offers a communication paradigm called unsolicited notification. When unsolicited notification occurs, a client receives a message that it has never requested. Using broker-less approach the Subscriber must use a filtering mechanism to process only messages it is interested in.
In case the broker-less approach over the UDP is selected for communication some multicast functionality must be offered by the protocol stack. UDP is one-to-one connectionless protocol and cannot be used for this purpose. The specification recommends using Internet Protocol (IP) multicast option to fulfill this requirement. Formally there are no additional mapping rules defined for this protocol, but as a result, this concrete protocol has been selected as the base for User Datagram Protocol (UDP) protocol and is an embedded part of the OSI Network Layer.
This approach has some drawbacks. Using IP multicast for UDP mapping, special equipment and dedicated configuration of that equipment are required. Both make this solution applicable only for the local network segments in the administration realm of the protocol users. It is hard to imagine the usage of this communication option even in case of enterprise scoped networks. From the practice, we know that particularly with factory networks, the manufacturing/engineering and IT organizations of the same company don't agree upon the management boundaries in a single plant. On the other hand, a broker-less PubSub UDP mapping using unicast addressing is a highly specialized case where the Publisher is intimately coupled to the Subscriber.
Every specification should promote interoperability. Unfortunately, Part 14 PubSub doesn't specify how a multicast address is acquired by a publisher and subscribers - but this is absolutely crucial to obtain interoperability. The core concept of the publisher/subscriber communication pattern is topic-based messages distribution. It seems difficult or even impossible to create any directory services based on the IP addressing mechanism because it addresses issues related to nodes identification and localization on the global network, but not data semantics (data meaning).
Detailed description of the UDP mapping rules are covered by the section Underlying Transport over UDP.
I guess that the removal of the UDP and IP protocols from the communication stack is recognized by the specification authors as a mean to improve the performance of the communication. As a result Ethernet mapping rules have been defined (see figure above). The Ethernet term is recognized as a keyword with a very broad meaning (IEEE 802.3 ETHERNET WORKING GROUP). The specification doesn't define normative reference in this respect. In the figure above it is presented as a concrete implementation compliant with the IEEE 802.3 standard suit. In case the UDP protocol is removed form the stack to replace the application selection functionality offered by the socket concept the registered B62C EtherType is recommended, which is used as the protocol discrimination. Removing IP from the communication stack means that the addressing possibility is limited to local network segment. Detailed description of the Ethernet mapping rules are covered by the section Underlying Transport over Ethernet.
NOTE 3: The specification doesn't delegate any publish/subscribe functionality down to this protocol. In this case, even the document title PubSub is confusing.
NOTE 4: Ethernet mapping is applicable only in case the communication parties are connected to the same local network segment (they are in the same broadcast domain limited by a VLAN if any).
Further communication performance improvement and extension of the functionality may be obtained for example by applying implementation of the 802.1Q-2018:
Time-Sensitive Network (TSN)
Virtual Local Network (VLAN)
Quality of Service (QoS)
They are only partially mentioned in the specification but the solution like these should be recognized and modeled as an embedded part of the abstract OSI Data Link Layer. In any case, these solutions are invisible for the implementation of the communication layers above OSI Data Link Layer, so they are invisible for upper layers and doesn't have any impact on the PubSub interoperability, therefore should be considered as statements outside the scope of the specification. It is also worth stressing that these solutions can be applied in spite of the above communication stack selection - it is the common point in the transport protocol stack for all mappings. In other words, the mentioned solutions are not dedicated to OPC UA at all and can be applied for any communication protocol.
The syntax and semantics of the messages exchanged between the PubSub Application network nodes are described as the NetworkMessage data structure. Each NetworkMessage includes header information (e.g. identification and security data) and one or more DataSetMessage structures. The DataSetMessage may be signed and encrypted in accordance with the configured message security. Each DataSetMessage contains process data.
The NetworkMessage structure can be serialized using the following encoding:
UADP: optimized binary encoding
JSON: text format as defined in RFC JSON
The following documents, in whole or in part, are normatively referenced in this document and are indispensable for its application.
The UAOOI.Networking.SemanticData library is designed to be a foundation of developing application programs that are taking part of message-centric communication pattern and interconnected using the reactive networking concept described in the section Semantic-Data Processing Architecture. To promote interoperability this library is a collection of types aimed at implementation of the Part 14 PubSub standard.
The preliminary code help documentation is available for sponsors- consider joining.
The topics contained in the document Getting Started Tutorial are intended to give you quick exposure to the OOI Reactive Application network-based data exchange programming experience. Working through this tutorial gives you an introductory understanding of the steps required to customize existing OOI Reactive Application.
The section Walk-through ReferenceApplication covers the description of a project aimed at implementation of an example of the OOI Reactive Application supporting producer and consumer roles simultaneously implemented as independent concurrent threads. The purpose of the ReferenceApplication is to demonstrate the concepts and architecture of the reactive networking application implementation, rather than to necessarily provide a realistic scenario for its use. For more extensive examples, see the Semantic-Data Processing Architecture.
I would like to thank Gary Workman, Michał Morawski, Michel Condemine, Jayachandran Rameshbabu, and Stéphane Potier for their feedback, cooperation and of course friendship.
Publish-subscribe is a messages distribution scenario where senders of messages, called publishers, do not send them directly to specific receivers, called subscribers, but instead categorize published messages into classes without knowledge of which subscribers if any, there may be. Similarly, subscribers express interest in one or more classes and only receive messages that are of interest, without knowledge of which publishers, if any, there are. In the publish-subscribe model, subscribers typically receive only a subset of the total messages published. The process of selecting messages for reception and processing is called filtering. There are two common forms of filtering: topic-based and content-based.
Data exchange scenario that requires a session connection be established before any data can be sent. Connection-oriented services set up virtual links between applications through a network. The session is responsible to retain a state information or status about each communicating partner for the duration of multiple requests. An OPC UA Client/Server connection is a stateful connection because both systems maintain information about the session itself during its life.
Messages exchange scenario that does not require a session connection between sender and receiver. The sender simply starts sending packets (called datagrams) to the destination. Neither system must maintain state information for the systems that they send messages to or receive messages from.
