Open Multimedia Tools

The “Open Multimedia Tools” project survies and investigate tools and technologies within the multimedia domain. A trend towards more openness in software solutions for multimedia applications as well as dcreasing cost for high quality multimedia hardward enable new use of multimedia in education and at new scales.

Among topics addressed by this project are
– Streaming
– Multimedia collaboration (high quality videoconferencing, web-meetings)
– Audio challanges
– Search and management of multimedia content

Results, being knowledge, reports and/or demonstrator tools, are likly to be applicable for the UNINETT eCampus program.

Trends in multimedia domains

Based on the experiences for the concluded “Open multimedia tools 2014” project, a list of seen trends and interesting technological development directions are presented.

  • Real-time collaboration: The advent of the WebRTC/rtcweb standards has already enable new browser base collaboration tools, without the need for plug-ins or other additional installed software in clients. It seems though that the standards aim for medium to low audio-visual quality. They will support negotiation for high quality media streams, but it remains to be seen if any of the popular browser will implement support for such streams. The none-commercial musical collaborations tools applied by UNINETT during experimentation lacks standardized signalling support. A potential improvement of these openly available tools is to add WebRTC based signalling and media transport. However keeping end-to-end delay to a minimum may turn out to be challenging. Good news with respect to delay is the growing availability of fast cameras, displays and projectors. Popular consumer interface standards like HDMI, DVI and USB now support bit-rates which enable image and audio capture at such high frame-rates, frequencies and resolutions that the inevitable buffering in the devices goes close to unnoticed.  Consumer standards also imply consumer products which again imply lower investment cost.
  • Multimedia search: Search in multimedia content is very much about extracting and/or adding text based meta-data to the multimedia objects. Most search engine index text-only efficiently. Extracting meta-data manually is time consuming, but results in high quality data. Extracting meta-data automatically is complex and computationally demanding, and often results in low quality data. A lot of research is ongoing in the multimedia meta-data extraction and classification domain, where deep learning (by neural networks) is a promising area. The big could-service players (Google, Facebook, Amazon, …) have already running systems with reasonable extraction accuracy, however these systems are not straight forward to access and may not be utilized in all contexts due to for instance privacy concerns. Content ownership has turned out to be a challenge with respect to search functionality for UNINETT’s video collections. Managing  authorization and access lists for content is and will continue to be a challenge since UNINETT’s users by default own all content they submit to the UNINETT multimedia storage facilities (i.e. lecture capture systems). Encouraging user to share content openly is an option, but care must be taken with respect to third party content included in the users content. Search systems which handle meta-data extraction as well as copyright issues in automated, efficient and trustworthy ways are still to be made available.
  • Streaming infrastructure: Video and audio streaming has been around for a decade and infrastructure technology has converged towards chunk based unicast streaming over network transport protocols (HTTP/TCP).  The “Dynamic Adaptive Streaming over HTTP” (DASH) standard has been available since 2011, a standard developed based on earlier protocols from Adobe (RTMP), Apple (HLS) and Microsoft (Silverlight).  Unicast streaming has obvious scalability issues and has contributed to the rise of content delivery network (CDN) providers (e.g. Akamai, Level3) which provide the necessary server infrastructure to fan-out the unicast streams. Unicast DASH-like streaming of to-days encoded multimedia content works well in wireless access networks since content recovery mechanisms for unicast flows are operational. However, during live streaming events where many users within the same wireless domain request the same stream, known as “flash crowds” (e.g. sport events, conferences, news broadcasts), unicast applied for broadcasting struggle to utilize network resources, which often lead to long delays and poor quality of delivered content. As client devices continue to improve their ability to consume high quality video and audio (as well as improve in capturing at high quality) more efficient utilisation of wireless resources is very much desirable. Multicast technology is in principle promising, but has to-day still many issues related to current media encoding standards and wireless access point functionality. Peer-to-peer live streaming (P2PLS) is on a rise, supported by the fresh WebRTC standard. P2PLS can reduce the need for CDN support, but will not likely help much with “flash crowd” challenges. Hence live streaming of high quality multimedia content in crowded settings is still to be handled properly in the future.

“Open multimedia tool 2014” ends – Results

The “Open Multimedia Tools 2014” project came to an end at the end of 2014. A summary of activities and achievements is presented  in this post.

  • High quality streaming / Musical collaboration: This activity has been looking into state-of-the-art tools for “near natural” real-time audio-visual online collaboration. It  has turned out to be the most fruitful activity in the project. A collection of tools have been investigated. They have been applied in several scientific experiments where the level of quality required to reach “near natural” quality has been investigated. Milestone for the activity are the following
    • Conducting of a choir over the internet successfully achieved [1]
    • Msc thesis completed titled “Telepresence Quality” [2]
    • Phase I of establishing infrastructure for musical collaboration completed [3]
    • A demo performance utilizing musical-collaboration infrastructure was given [4]
  • Search in multimedia: This activity has continued previous work on a experimental search engine for multimedia content. A new renovated and enhanced version of the edusearch.uninett.no test service was produced and to some extent operational. Unfortunately due to security issues with the server hosting the service, it is currently off-line.
  • Streaming infrastructure: A test service for live video streaming base on the Wowza streaming server software has been kept operational until end of 2014.  The service was decided not to be further developed as UNINETT’s medaisite-based lecture recording service also supports live streaming and will support input from general IP media devices in the near future. Two Msc specialisation projects from NTNU investigated investigated multimedia distribution aspects in UNINETT, one addressed the potential and limits of more active use of multicast [5] while the other analysed the potential of introducing CDN nodes in UNINETT [6]. (Note: The reports are from student projects, hence quality is not guaranteed.) A Msc thesis on multicast challenges in wifi access networks was also completed [7].
  • Advanced audio: One Msc specialisation project address the potential for doing audio processing in to-days browsers [8]. Promising results showed that audio sources in a stereo audio signal could be detected and filtered, i.e. sound sources in a “raw” lecture recording could hence potentially be streamed unprocessed to a browser where processing could be tuned by the user and his needs.
  • Research applications: During the project UNINETT participated in consortia for several multimedia-related application to the EU FP7. Unfortunately, none of the applications were approved.

Some of the activities in  “Open Multimedia Tools 2014” will be continued in the upcoming “Multimedia Infrastructure 2015” project.

 References

  1. Conducting over IP
  2. Telepresence Quality“, Daniel Puig Conca, Msc thesis, NTNU, 10. jun 2012.
  3. Nettmusikk web-site
  4. Thora“, Dokkhuset Scene, Dec 09, 2015
  5. “Multicast capacity estimated for Lecture IP TV in UNINETT”, Sepideh Kanani, Msc specialization project, NTNU, July 2013.
  6. “Content Delivery Networks in UNINETT”, Haakon Løchen Nore, Msc specialization project, NTNU, Dec 2013.
  7. An investigation of large scale challenges with live video streaming over Wi-Fi access networks“, Sepideh Kanani, Msc thesis, May 2014.
  8. “Audio processing in HTML5”, Jonathan Klapel, Msc specialization project, NTNU, Dec 2013.

 

 

Demo performance the opera “Thora” at Dokkhuset

The “Open Multimedia Tools 2014” project concludes this fall. As a final delivery from the project a demo performance will take place at Dokkhuset scene December 9th. A digest of the opera “Thora” will be presented with the intention to show how newly installed audio-visual technology may be utilized in a live performence.

The sub-project “Nettmusikk 2014” has brought Dokkhuset scene and NTNU together in a process to conclude the long running scientific project “Collaboration spaces”, a collaboration between the three NTNU departments Telematics, Music and Electronics and Telecommunications. Close to 2 million NOK has been invested in audio-visual equipment and 10 networking, upgrading Dokkhuset, Orgelsalen and two studios at the department of Music.

UNINETT has contributed with project management, technical resources on networking and networked performance tools as well as some funding of equipment, installation and testing. NTNU has been the main source of funding with 1.8 million NOK invested.

“Nettmusikk” will most likely be continued in 2015 as more institutions (UiT, UiA and NMH) are already very interested in joining the networked performance infrastructure.