Video chat side channel in a remote class (or an experiment not good enough to keep)

Talking about failure is hard. Talking about an experiment that didn’t completely fail but didn’t wholly succeed, is something that I could easily “sweep under the rug.” However, I think it’s important to share the good, the bad, and the ugly. It shows we are human and no one is perfect. Moreover, for this particular experiment, I could see myself rationalizing trying again, and I want to make sure I remember the lessons I learned.

In Spring 2021, when we were teaching our CS2 class fully remote due to covid-19, I had a crazy idea. What if we gave students a second video chat outside the lecture Zoom meeting? What if we used this video side channel to simulate the “talk to your neighbor” practice that I used a lot in my lecture teaching? It did not go as well as I had imagined, and this blog post is a reflection on why we did it, how things went, and what we did when things weren’t working.

[This article is also posted on my personal blog so it is freely accessible.]

Why we did it

I firmly believe discussing new concepts helps with learning. Having students predict the output of code or answer peer instruction questions helps students see what they actually know versus what they only think they understand. Layering discussion with their peers on top helps because it requires not only picking an answer but articulating their reasoning. Moreover, if they are confused, their peer, who also just learned it, can help explain it in a way that is at that student’s level compared to someone suffering from expert’s fallacy.

Despite lecturing over zoom, I wanted to try to enable student peer discussion. However, while zoom breakout rooms can facilitate small discussion, they take 3–5 minutes to transition between. Mix in the desire to have students discuss 3–5 times a lecture and that’s 9–25 minutes lost in transitioning from a 75-minute class!

So my not so brilliant idea was to create a second channel through the Microsoft Teams platform, which was already integrated with classes here at Duke. With this side-channel, there would be no transition time because the students would have both during class. Moreover, they didn’t have to use the video. It was a channel that had text, voice, or video.

I will acknowledge the privilege that came with this idea. I work at a private school, so I had reasonable confidence that all of my students had a device and internet connection that could handle being on Zoom and this side channel. I also believed that, at minimum, the students would have a text chat through Teams.

How we did it

We started the semester using random zoom breakout rooms for discussion. However, on the first day, I told the class we would transition to the side channel after a few weeks (when class add/drop was over and we’d figured things out). I introduced the idea as an experiment and emphasized that it was both worth trying and experiments were allowed to fail. I framed the motivation as to give them “neighbors” to talk to while in class and not just feel like one among a faceless many.

The class’s outstanding teaching associate, Kate, handled creating each smaller team in the class’s Microsoft Team. We clustered students based on their discussion section and time zone. We hoped it would add to the continuity of interactions and help build a community. The undergrad TAs were in charge of notifying the students in their discussion section which small team the student was in.

What happened

We asked the students about the side-channel during our mid-semester survey (which most students respond to). We had Likert scale strongly agree to strongly disagree questions on if they thought it was a good idea, if it was well executed, and if they wanted to continue the side channel. 64% of the respondents agreed to some degree that it was a good idea, with another 23% neutral. However, fewer thought it was well executed at 45%. 27% were neutral and 28% disagreed, so most likely, those that thought the idea was good didn’t think we executed the idea very well. As for keeping it, less than a majority agreed to keep it (45%), while 25% were neutral, and 30% wanted things to change.

After the Likert scale questions, we asked them what they wanted to do, with the options: (1) random breakout rooms, (2) assigned groups (our current system), and (3) self-organized groups outside of the systems we had in place. A majority wanted breakout rooms.

Finally, to better understand why the experiment wasn’t going well, we asked them if they had trouble talking to their group and why. 40% said they had no trouble. Another 32% said they were taking the class asynchronous and therefore didn’t attend class. 3% actually said they didn’t want to talk to anyone. In contrast, 25% of respondents had trouble talking to their group, mainly due to a lack of response from other group members.

So we ended the experiment. 25% of the class having no interaction was too high to make continuing the current system worthwhile. And that is how we framed it to the class. We spent the rest of the semester using random breakout rooms. At the time, it was a tough call. 40% of the class were talking to their peers, and 45% thought the system was working. The number was just not high enough to be worth the cost of the students not benefiting from the system.

Conclusion

More than a semester later, and reflecting on what we did, I don’t think I would have changed any of it. I think our decision process was sound, though, at the time, it felt like we were making things up as we went. And I’m glad that we held to the process because, honestly, it would have been easier to just pretend things were fine and not measure how things were actually going. Then we would have found out at the end of the semester from the course evaluations that it wasn’t benefiting enough students, and we just regret the ones that weren’t benefiting. This process got us to change course mid-way and attempt to improve things.

One thing I regret is not systematically collecting at the end of the semester whether the mid-semester change improved things. Some students volunteered this information in emails soon after the decision and in the open-response questions of the course evals. And all those comments were positive. They appreciated we were willing to listen and change things based on the student responses. But the scientist in me knows that is a biased sample.

It’s still frustrating that the experiment wasn’t a “slam dunk” as a bad idea or a good idea. If it had been a clearly terrible idea and benefited no one, I wouldn’t have the guilt of knowing I was stopping a practice that was helping some of the students. But if the goal is to have the greatest benefit and the least harm, then we made the right decision.

Hopefully, sharing this is helpful to someone. And if anyone would like to learn more of what we did or would like to share their own experiences, feel free to reach out.