8 Aug’16

Azure and Office 365Introduction to Azure Event Hubs

Overview

  1. Microsoft Azure Event Hubs is a managed platform service that can intake a large amounts of data for various scenarios.
  2. It is a highly scalable, low latency and highly-scalable data ingest system. Data is ingested here in the form of events.
  3. Event Publishers submit data to the Event Hub and Event Consumers consume the data at their own time.
  4. Some scenarios where Event Hubs are applicable are – Application Instrumentation, user experience, Internet of Things (IoT)
  5. Event Hubs reside in the Service Bus namespace.
  6. Event Hubs uses AMQP and HTTP as its primary API interfaces.

  Below diagram will give a high level overview of where Event Hubs lie:

 

Partitions in Event Hubs

  1. Partitions are ordered sequence of events that reside in the event hubs. Newer partitions are added to the end of the queue as they arrive.
  2. Partitions retain data for a configured period of time. This setting is common across all partitions in the Event Hub.
  3. Every partitioned is populated at their own pace and not necessarily sequentially. Hence, data in partitions grow independently.
  4. of partitions are specified while creation of Event Hubs. This number should be between 2 and 32. Default partitions allotted are 4.
  5. The number of partitions you choose are more related to the number of concurrent consuming applications you expect to have.
  6. This partition count cannot be changed once the event hub is created.

 

Event Publishers

So who are event publishers?

  1. The entities that populates data to the event hubs are the event publishers.
  2. Event publishers can publish data to the event hubs either using HTTPS or AMQP 1.0.
  3. Event publishers use a SAS (Shared Access Key) token to authenticate themselves to Event Hubs.

 

Common Tasks for a Publisher

Acquire an SAS Token

  1. SAS is the authentication mechanism for Event Hubs. Service Bus provides SAS policies at the namespace level and at the Event Hub level.
  2. Service Bus can regenerate the key and authenticate the sender.

 
Publishing an Event

  1. Service Bus provides an EventHubClient class for publishing events to an Event Hub from .NET clients.
  2. Events can either be published individually or batched.
  3. A single publication has a limitation of 256 KB, whether batch or individually. Publishing events larger than this will result in error.

 
Partition Key

  1. It is a value to map incoming messages to the specific partitions for data organization purpose.
  2. This is a sender supplied value passed to the event hub.
  3. It is processed through a hashing function which creates the partition assignment.
  4. Partition Keys are important for organizing data for downstream processing.

Below Diagram explains how Partition Key work:

 

Event Consumers

  1. An entity that reads event data from Event Hub is an event consumer.
  2. All event consumers read from the partitions in the consumer group.
  3. Each partition should only have one active reader at a time.

 

Consumer Groups

  1. A consumer groups is a view (state, position, or offset) of the entire Event Hub.
  2. Consumers groups lets consuming applications have a separate view of the entire Event Hub.
  3. There is always a default consumer group. You can create up to 20 consumer groups in an Event Hub.

 

Stream Offsets & Checkpointing

  1. An offset is a position of an event within a partition. It is a client-marker to specify at which point should the processing should happen from.
  2. Consumers should store their own offsets.
  3. Checkpointing is a process where readers mark their position in a partition in the event hubs.

Common Consumer Tasks

  1. All consumers connect to the Event Hub via AMQP 1.0. It is a session and state-aware bidirectional communication channel.
  2. As a partitioned consumer model, only 1 consumer can be active on a partition at a time within a consumer group.
  3. The following data is read from the Event Hub
    1. Offset
    2. Sequence Number
    3. Body
    4. User Properties
    5. System Properties.
  4. As mentioned above, it is user’s responsibility to maintain this offset.
  5. So now you know about Event Hubs!

Summary

Azure Event Hubs provide high-scalable, telemetry processing service that can be used for common applications. Event Hubs provide low latency.

In the next part of the blog, I’ll be covering a technical look at the Event Hubs wherein Dynamics CRM will publish data to the Event Hubs and how data is available for applications to consume! Watch out here for the upcoming blog very soon! Hope this overview was helpful.

 

Written by

Team Member

CloudFronts

Leave a Reply

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload the CAPTCHA.

Want to streamline your business processes?

  • This field is for validation purposes and should be left unchanged.

Recent Articles

  • Reading more then 10K records in D3FOE OData API

    11 July’ 2018

    Introduction: We all know Dynamics 365 Finance and Operations has limitation of 10K records to be fetched at a time usi...

    Read more
  • Paging in D365 Customer Engagement v9.0

    10 July’ 2018

    Introduction: The Xrm.retrieveMultipleRecords method is used to retrieve a collection of records in Dynamics 365 Custom...

    Read more
  • Set up Dynamics 365 connection in Microsoft Social Engagement

    10 July’ 2018

    Introduction: This blog explains how to Set up Dynamics 365 connection in Microsoft Social Engagement. Steps to be follo...

    Read more
  • Voice of the Customer failed to install

    10 July’ 2018

    Introduction: Many people face issues in installing Voice of Customer solution on v9 environment and trying repeatedly ...

    Read more
  • Scribe Insight AX as a Web Service Find Block issue

    10 July’ 2018

    Introduction: If we need to look up for any value from AX then we do it by using a Find Block in Scribe Insight Eg: Basi...

    Read more