Connecting Microsoft Teams Status to an IOT device

I need not explain about the remote work situation now. At home my family has somehow adjusted to this remote working. Like everyone I use a shared room as home office and lock it during office time for some focused work. When someone wants to access the room, they knock the door. If I am free then I open the door or else I don’t open the door. My family members never know when I am in meeting and when I am free. Only way to find out is knock the door. At times I get irritated by this interruption. One day my spouse mentioned, why don’t you put light outside your door about your status.

There is a saying when a software engineer gives an estimate, it is always an under estimation. Because sometimes a simple thing becomes really complicated. I too did something similar. let me tell you the story. To start with, my mental solution ran something like this

  • Teams should expose some kind of automation API.
  • Use the API and fetch the status information.
  • Based on the status, update a led connected to Nodemcu IOT device.

And I estimated to finish this during Friday evening.

Teams Status is part of MS Graph API

After some search I found the “Status” information is part of Graph API. And Presence information is in beta state as of now. I gave a try in the Graph Explorer

https://graph.microsoft.com/beta/me/presence

And I received the response as

{
"@odata.context": "https://graph.microsoft.com/beta/$metadata#users('<user id guid>')/presence/$entity",
"id": "<user id guid>",
"availability": "Busy",
"activity": "InACall",
"outOfOfficeSettings": 
{
"message": null,
"isOutOfOffice": false 
}
}

I was happy to see the information available in the reply JSON. Now all that I need to do, is to poll this api and update my status led.

Push vs Poll status information

I felt a bit bad about polling the status information, since I have to decide on a frequency. If I need the update very fast(near real time) I need to poll frequently and on the other hand if I don’t want to bombard the polling API endpoint (or even get blocked by the Graph API endpoint) I should reduce the frequency. My developer spider sense told me that there should be some kind of push notification for this information. As expected Graph API provided a pub-sub model change notification for this case. I can add a subscription using this endpoint

https://graph.microsoft.com/v1.0/subscriptions

and POST a message in JSON format

{
     "changeType": "updated",
     "notificationUrl": "https://<mywebapi>.azurewebsites.net/api/TeamStatusHook",
     "resource": "/communications/presences/<user id guid>",
     "expirationDateTime": "2021-08-13T22:00:00.0Z",
     "clientState": "FeroseStatusUpdate",
     "latestSupportedTlsVersion": "v1_2"
}

and received the successful subscription for an hour(for some reason they give the subscription only for an hour always) as below.

{
"@odata.context": "https://graph.microsoft.com/v1.0/$metadata#subscriptions/$entity",
"id": "22fdb7f7-15ce-4568-b3fb-47e0aaa8db28",
"resource": "/communications/presences/<user id guid>",
"applicationId": "de8bc8b5-d9f9-48b1-a8ad-b748da725064",
"changeType": "updated",
"clientState": "FeroseStatusUpdate",
"notificationUrl": "https://<mywebapi>.azurewebsites.net/api/TeamStatusHook",
"notificationQueryOptions": null,
"lifecycleNotificationUrl": null,
"expirationDateTime": "2021-08-16T06:36:02.1704178Z",
"creatorId": "<user id guid>",
"includeResourceData": null,
"latestSupportedTlsVersion": "v1_2",
"encryptionCertificate": null,
"encryptionCertificateId": null,
"notificationUrlAppId": null
}

Setting up the Call back URL

In the above subscription you can see I gave

https://<mywebapi>.azurewebsites.net/api/TeamStatusHook

as the call back URL. As per the documentation this end point should be capable of responding to two kind of requests.

  1. Graph system shall send POST with a validation token as query parameter and content type text/plain. the end point shall respond within 10 secs
    1. The end point shall return the token as a body,
    2. Content type as text/plain
    3. HTTP 200 OK.
  2. Graph system shall send the real update as below. And the end point shall respond with a 201 Accepted.
{
"value":
         [
             {
                 "subscriptionId":"e96cf426-e71a-47b5-9a48-52b17a3e3b27",
                 "clientState":"FeroseStatusUpdate",
                 "changeType":"updated",
                 "resource":"communications/presences('<user id guid>')",
                 "subscriptionExpirationDateTime":"2021-08-13T06:06:58.660307-07:00",
                 "resourceData":
                 {
                     "@odata.type":"#Microsoft.Graph.presence",
                     "@odata.id":"communications/presences('<user id guid>')",
                     "id":"<user id guid>",
                     "activity":"Busy",
                     "availability":"Busy"
                 },
             "tenantId":"<tenant id guid>"
             }
         ]
}

I thought of setting up an Azure Function, but I have to store the update for my IOT device queries. Setting up a stateful Azure Function was a bit complicated, so I went with a ASP.Net core web api app. As a shortcut I stored the status in a static variable of web api controller.

[Route("api/[controller]")]
[ApiController]
public class TeamStatusHookController : ControllerBase
{
     static string currentStatus = "{\"Status\":\"Unknown\"}";

    [HttpPost]
     public IActionResult UpdateStatus()
     {
         var contentType = HttpContext.Request.Headers["Content-Type"];
         if (contentType.ToString().Contains("text/plain"))
         { 
             var validationToken = HttpContext.Request.Query["validationToken"].ToString();
             return Content(validationToken);
         }
         if (contentType.ToString().Contains("application/json"))
         {
             using (StreamReader reader = new StreamReader(Request.Body, Encoding.UTF8))
             {
                 var jsonData = JsonDocument.Parse(reader.ReadToEndAsync().Result);
                 var activity = jsonData.RootElement.GetProperty("value")[0].GetProperty("resourceData").GetProperty("activity");
                 var availability = jsonData.RootElement.GetProperty("value")[0].GetProperty("resourceData").GetProperty("availability");
                 currentStatus = $"{{\"availability\":\"{availability}\",\"activity\":\"{activity}\"}}";
             }
             return Accepted();
         }
         return NotFound();
     }

    [HttpGet]
     public JsonResult GetStatus()
     {
         return new JsonResult(currentStatus);
     }

It worked and I can see my status information in the GetStatus endpoint.

Connecting the IOT device

The final step was to connect the information to the IOT device. I had a nodemcu (programmable ESP8266) lying around with an in built led. I connected this to the status check api. I polled every 5 seconds (to see how things work).

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>
#include <ESP8266HTTPClient.h>
#include <WiFiClientSecureBearSSL.h>

/* Set these to your desired credentials. */
const char *ssid = "<ssid>"; //Enter your WIFI ssid
const char *password = "<passwd>"; //Enter your WIFI password

ESP8266WiFiMulti WiFiMulti;

void setup() {
   pinMode(LED_BUILTIN, OUTPUT);
   Serial.begin(115200);

  Serial.println();
   Serial.println();
   Serial.println();

  for (uint8_t t = 4; t > 0; t--) {
     Serial.printf("[SETUP] WAIT %d...\n", t);
     Serial.flush();
     delay(1000);
   }

  WiFi.mode(WIFI_STA);
   WiFiMulti.addAP(ssid, password);
   
}

void loop() {
   // wait for WiFi connection
   if ((WiFiMulti.run() == WL_CONNECTED)) {

    std::unique_ptr<BearSSL::WiFiClientSecure>client(new BearSSL::WiFiClientSecure);

    //client->setFingerprint(fingerprint);
     // for now disable https certificate check:
     client->setInsecure();

    HTTPClient https;

    Serial.print("[HTTPS] begin...\n");
     if (https.begin(*client, "https://<mywebapi>.azurewebsites.net/api/TeamStatusHook")){
      // HTTPS
      Serial.print("[HTTPS] GET...\n");
       // start connection and send HTTP header
       int httpCode = https.GET();

      // httpCode will be negative on error
       if (httpCode > 0) {
         // HTTP header has been send and Server response header has been handled
         Serial.printf("[HTTPS] GET... code: %d\n", httpCode);

        // file found at server
         if (httpCode == HTTP_CODE_OK || httpCode == HTTP_CODE_MOVED_PERMANENTLY) {
           String payload = https.getString();
           if(payload.indexOf("Busy")>0 || payload.indexOf("DoNotDisturb")>0){
                digitalWrite(LED_BUILTIN, 0);
           }else{
                digitalWrite(LED_BUILTIN, 1);
           }
           Serial.println(payload);
         }
       } else {
         Serial.printf("[HTTPS] GET... failed, error: %s\n", https.errorToString(httpCode).c_str());
       }
      https.end();
     } else {
       Serial.printf("[HTTPS] Unable to connect\n");
     }
   }
  Serial.println("Wait 5s before next round...");
   delay(5000);
}


Again things worked fine. I can see the status led light up when I am busy. Now it is time to move this solution to production and sustain.

aebff969-6886-499b-a916-b3c35aae6fbaea396ae1-4622-4b92-b217-16f652548e33

Move the application to production

First step is to subscribe for this change notification from the web app. And I need an access token to invoke the endpoint. To get an access token you need a client id and client secret. And for that you need app registered in Azure AD. I tried registering the app, unfortunately the app registration is blocked by office 365 admin for security reason. I panicked a bit. ok let me try it in my other azure subscription, the app registration went through this time without any issue, but I cannot add a subscription for my teams tenant from this app (basically the app has to be approved by the teams admin again). I tried using power automate for the subscription. But invoking a REST API is premium and not available to me under the free plan. Slowly my options ran out one by one. The only option left was to create/update the subscription manually in the graph explorer. This is not a viable option for me. And I can sense the failure here.

You don’t own your data

Until now when I did something with an VBA office automation (Word/Excel), I owned the data in my machine or easily can access it. Now MS Teams being a cloud app, the teams status updates beautifully in the desktop app, web site, iPhone app and android tablet but not in my toy app. I realized that I don’t own my teams data anymore. Or I own the data, but cannot easily access that with my own toy app anymore. I need the app cleared by office 365 admin.

I wrote this down so that someone who has access to the final step can setup a IOT status led.

Update: Corrected some code that was corrupted by wordpress editor last time.

Subtitles from Azure AI

I started a YouTube channel  couple of weeks back without knowing what I am getting into. For a long time I was planning to upload some training videos to YouTube. I was stuck in the analysis paralysis. One day I took the plunge and added my first training video on the single responsibility principle. After publishing I learned the importance of subtitles.

YouTube subtitle editor.

Some of my close friends asked for it. So I started to create the subtitle manually. This was a tedious and painful job. Because I am not so good at typing. I couldn’t keep up with the video’s pace. For a 20 mins video I spent 60 mins and had covered only 10 mins of transcription. At this speed I would need 3 more hours to complete the subtitle process. I stopped for a moment and started to explore other options.

  • Use iphone’s speech to text typing. But somehow this worked for my voice not the audio from speaker.
  • Ask someone to transcribe the video for me. I have to find someone for this task. Someone who could understand my speech.
  • See if azure or aws provides a solution for this task.

Speech to Text from Azure Cognitive Services

Azure provided a speech to text as a part of azure cognitive services. I was bit sceptical because most of the speech to text services work for native English speakers but not the others. Anyway I decided to give a try. I took the sample application as is. I have removed some parts here to fit the code inside the blog. But this should work for you.

class Program
{
async static Task FromFile(SpeechConfig speechConfig)
{
	var audioConfig = AudioConfig.FromWavFileInput(@"D:\audio.wav");
	var recognizer = new SpeechRecognizer(speechConfig, audioConfig);
	var stopRecognition = new TaskCompletionSource();

	recognizer.Recognized += (s, e) =>
	{
		if (e.Result.Reason == ResultReason.RecognizedSpeech)
		{
			Console.WriteLine($"{e.Result.Text} ");
		}
	};

	recognizer.Canceled += (s, e) =>
	{
		Console.WriteLine($"CANCELED: Reason={e.Reason}");
		stopRecognition.TrySetResult(0);
	};

	recognizer.SessionStopped += (s, e) =>
	{
		Console.WriteLine("\n    Session stopped event.");
		stopRecognition.TrySetResult(0);
	};
	await recognizer.StartContinuousRecognitionAsync();
	Task.WaitAny(new[] { stopRecognition.Task });
}

async static Task Main(string[] args)
{
	var speechConfig = SpeechConfig.FromSubscription(key,region);
	speechConfig.SpeechRecognitionLanguage = "en-IN";

	await FromFile(speechConfig);
	Console.ReadLine();
}
}



This small program takes a wav file and converts the speech to text. Only change that I had to do was change the language to en-IN in the speech Config. The speech to text was 90% accurate. I was stunned when I ran the program for the first time. In the past for speech to text, I had to train the model myself reading some complicated English passages.  Now with the advent of pre-trained Cloud AI this cumbersome training step is gone. I use this for all my videos now. It works like a charm.

And one limitation as of now is the free version takes wav file as input. I use our trusted ffmpeg to get audio as a  wav file.

ffmpeg.exe -i '.\2021-08-05 19-31-19.mp4' audio.wav

Hope this helps someone. Good job Azure guys.