Search This Blog

Sunday, 31 July 2011

Using VLCJ for video reliably with out of process players

This post is really a follow on to the last, so if you want a bit of background give that a read through. In it I basically talk about the various options I came to when trying to implement video support in Java, and how I felt the best way to go was using out of process players and VLCJ.

Before I get to the code, a warning - while this isn't stupidly hard, it's definitely not for beginners. If you're competent with Java in general you shouldn't have any trouble following - but this is not a cut / paste / forget the advanced stuff job! Eventually and depending on interest I may look into packaging it up into an easy to use library and distributing, but that day is not today!

Secondly, this code is literally cut straight out of Quelea and at the moment is:

  • Largely uncommented

  • From an unreleased version

  • Not the best quality

  • May contain Quelea specific bits that don't apply


Eventually, it'll all be refactored into beautiful OO-like niceness. But in terms of the concepts, it should be enough to get them across.

An overview


Essentially, how it works is by firing off a separate process executing all the native VLC code. The references to the standard output and standard input streams of the "other" process(es) are saved, and these are used for communication between the processes. (It could just as easily be done by sockets, shared memory magic, some generic RMI framework and so on. But this for me is a scalable solution that just works without any substantial libraries or potential problems with firewalls that you can get with sockets.)

There's a rudimentary protocol that maps commands sent between the streams to the actions the out of process player should take, or the values it returns. The API user doesn't need to worry about these, since it's all encapsulated in an easy to use class, RemotePlayer.

I'm using this approach to run 3 concurrent media players in Quelea, and I've had 0 VM blowouts thus far in all the trial runs I've done (which is a fair number!) The guy in charge of VLCJ also reports the same behaviour when using out of process players. However you accomplish it, this seems undoubtedly the way to go if you want to prevent your application from crashing.

The actual code bit


So, now actually onto the code. Where it all starts from the user's point of view is RemotePlayerFactory:


package org.quelea.video;

import com.sun.jna.Native;
import java.awt.Canvas;

public class RemotePlayerFactory {

public static RemotePlayer getRemotePlayer(Canvas canvas) {
try {
long drawable = Native.getComponentID(canvas);
StreamWrapper wrapper = startSecondJVM(drawable);
final RemotePlayer player = new RemotePlayer(wrapper);
Runtime.getRuntime().addShutdownHook(new Thread() {
@Override
public void run() {
player.close();
}
});
return player;
}
catch (Exception ex) {
throw new RuntimeException("Couldn't create remote player", ex);
}
}

private static StreamWrapper startSecondJVM(long drawable) throws Exception {
String separator = System.getProperty("file.separator");
String classpath = System.getProperty("java.class.path");
String path = System.getProperty("java.home")
+ separator + "bin" + separator + "java";
ProcessBuilder processBuilder = new ProcessBuilder(path, "-cp", classpath, "-Djna.library.path=" + System.getProperty("jna.library.path"), OutOfProcessPlayer.class.getName(), Long.toString(drawable));
Process process = processBuilder.start();
return new StreamWrapper(process.getInputStream(), process.getOutputStream());
}
}


The call that might be most unfamiliar initially is the JNA one - Native.getComponentId(). Every heavyweight component has an ID which is assigned at the OS level and used for drawing to a particular window. It's this component ID that we'll be passing to the separate process, which it will use to draw to the window owned by the parent process. Notice also the shutdown hook so the other VM is terminated when this one is (that's essentially what the close method does, more on that later.) It's not a foolproof approach but it's good just in case it doesn't get cleared up otherwise.

In terms of the startSecondJVM method, this is a pretty standard, cross-platform (as much as I care about anyway, Windows, MacOS and Linux should all be fine) method to start up a second JVM. It starts the OutOfProcessPlayer with the component ID as its argument. There's just a few differences - firstly, we copy over the classpath and JNA library path of this VM so it can execute the class in this project which executes native VLC code without any problems. Secondly, it captures the streams in a StreamWrapper object, which is just as follows:



package org.quelea.video;

import java.io.InputStream;
import java.io.OutputStream;

/**
*
* @author Michael
*/
public class StreamWrapper {

private InputStream inputStream;
private OutputStream outputStream;

StreamWrapper(InputStream inputStream, OutputStream outputStream) {
this.inputStream = inputStream;
this.outputStream = outputStream;
}

public InputStream getInputStream() {
return inputStream;
}

public OutputStream getOutputStream() {
return outputStream;
}
}


Nothing special here, it's literally just wrapping up the input and output streams.

Onto the two core classes here, RemotePlayer and OutOfProcessPlayer. As the name suggests, the latter is the one that sits out of process.

RemotePlayer.java:


package org.quelea.video;

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;

/**
* Controls an OutOfProcessPlayer via input / output process streams.
* @author Michael
*/
public class RemotePlayer {

private BufferedReader in;
private BufferedWriter out;
private boolean open;
private boolean playing;
private boolean paused;

/**
* Internal use only.
*/
RemotePlayer(StreamWrapper wrapper) {
out = new BufferedWriter(new OutputStreamWriter(wrapper.getOutputStream()));
in = new BufferedReader(new InputStreamReader(wrapper.getInputStream()));
playing = false;
open = true;
}

private void writeOut(String command) {
if (!open) {
throw new IllegalArgumentException("This remote player has been closed!");
}
try {
out.write(command + "\n");
out.flush();
}
catch (IOException ex) {
throw new RuntimeException("Couldn't perform operation", ex);
}
}

private String getInput() {
try {
return in.readLine();
}
catch (IOException ex) {
throw new RuntimeException("Couldn't perform operation", ex);
}
}

public void load(String path) {
writeOut("open " + path);
}

public void play() {
writeOut("play");
playing = true;
paused = false;
}

public void pause() {
if(!paused) {
writeOut("pause");
playing = false;
paused = true;
}
}

public void stop() {
writeOut("stop");
playing = false;
paused = false;
}

public boolean isPlayable() {
writeOut("playable?");
return Boolean.parseBoolean(getInput());
}

public long getLength() {
writeOut("length?");
return Long.parseLong(getInput());
}

public long getTime() {
writeOut("time?");
return Long.parseLong(getInput());
}

public void setTime(long time) {
writeOut("setTime " + time);
}

public boolean getMute() {
writeOut("mute?");
return Boolean.parseBoolean(getInput());
}

public void setMute(boolean mute) {
writeOut("setMute " + mute);
}

/**
* Terminate the OutOfProcessPlayer. MUST be called before closing, otherwise
* the player won't quit!
*/
public void close() {
if (open) {
writeOut("close");
playing = false;
open = false;
}
}

/**
* Determine whether the remote player is playing.
* @return true if its playing, false otherwise.
*/
public boolean isPlaying() {
return playing;
}

/**
* Determine whether the remote player is paused.
* @return true if its paused, false otherwise.
*/
public boolean isPaused() {
return paused;
}

}


OutOfProcessPlayer.java:

package org.quelea.video;

import com.sun.jna.NativeLibrary;
import com.sun.jna.Pointer;
import java.awt.Canvas;
import java.io.BufferedReader;
import java.io.File;
import java.io.InputStreamReader;
import java.io.PrintStream;
import org.quelea.utils.QueleaProperties;
import uk.co.caprica.vlcj.binding.LibVlcFactory;
import uk.co.caprica.vlcj.binding.internal.libvlc_media_player_t;
import uk.co.caprica.vlcj.player.embedded.EmbeddedMediaPlayer;
import uk.co.caprica.vlcj.player.embedded.linux.LinuxEmbeddedMediaPlayer;
import uk.co.caprica.vlcj.player.embedded.mac.MacEmbeddedMediaPlayer;
import uk.co.caprica.vlcj.player.embedded.windows.WindowsEmbeddedMediaPlayer;
import uk.co.caprica.vlcj.runtime.RuntimeUtil;

/**
* Sits out of process so as not to crash the primary VM.
* @author Michael
*/
public class OutOfProcessPlayer {

public OutOfProcessPlayer(final long canvasId) throws Exception {

//Lifted pretty much out of the VLCJ code
EmbeddedMediaPlayer mediaPlayer;
if (RuntimeUtil.isNix()) {
mediaPlayer = new LinuxEmbeddedMediaPlayer(LibVlcFactory.factory().synchronise().log().create().libvlc_new(1, new String[]{"--no-video-title"}), null) {

@Override
protected void nativeSetVideoSurface(libvlc_media_player_t mediaPlayerInstance, Canvas videoSurface) {
libvlc.libvlc_media_player_set_xwindow(mediaPlayerInstance, (int) canvasId);
}
};
}
else if (RuntimeUtil.isWindows()) {
mediaPlayer = new WindowsEmbeddedMediaPlayer(LibVlcFactory.factory().synchronise().log().create().libvlc_new(1, new String[]{"--no-video-title"}), null) {

@Override
protected void nativeSetVideoSurface(libvlc_media_player_t mediaPlayerInstance, Canvas videoSurface) {
Pointer ptr = Pointer.createConstant(canvasId);
libvlc.libvlc_media_player_set_hwnd(mediaPlayerInstance, ptr);
}
};
}
else if (RuntimeUtil.isMac()) {
mediaPlayer = new MacEmbeddedMediaPlayer(LibVlcFactory.factory().synchronise().log().create().libvlc_new(2, new String[]{"--no-video-title", "--vout=macosx"}), null) {

@Override
protected void nativeSetVideoSurface(libvlc_media_player_t mediaPlayerInstance, Canvas videoSurface) {
Pointer ptr = Pointer.createConstant(canvasId);
libvlc.libvlc_media_player_set_nsobject(mediaPlayerInstance, ptr);
}
};
}
else {
mediaPlayer = null;
System.exit(1);
}

mediaPlayer.setVideoSurface(new Canvas());

BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
String inputLine;

//Process the input - I know this isn't very OO but it works for now...
while ((inputLine = in.readLine()) != null) {
if (inputLine.startsWith("open ")) {
inputLine = inputLine.substring("open ".length());
mediaPlayer.prepareMedia(inputLine);
}
else if (inputLine.equalsIgnoreCase("play")) {
mediaPlayer.play();
}
else if (inputLine.equalsIgnoreCase("pause")) {
mediaPlayer.pause();
}
else if (inputLine.equalsIgnoreCase("stop")) {
mediaPlayer.stop();
}
else if (inputLine.equalsIgnoreCase("playable?")) {
System.out.println(mediaPlayer.isPlayable());
}
else if (inputLine.startsWith("setTime ")) {
inputLine = inputLine.substring("setTime ".length());
mediaPlayer.setTime(Long.parseLong(inputLine));
}
else if (inputLine.startsWith("setMute ")) {
inputLine = inputLine.substring("setMute ".length());
mediaPlayer.mute(Boolean.parseBoolean(inputLine));
}
else if (inputLine.equalsIgnoreCase("mute?")) {
boolean mute = mediaPlayer.isMute();
System.out.println(mute);
}
else if (inputLine.equalsIgnoreCase("length?")) {
long length = mediaPlayer.getLength();
System.out.println(length);
}
else if (inputLine.equalsIgnoreCase("time?")) {
long time = mediaPlayer.getTime();
System.out.println(time);
}
else if (inputLine.equalsIgnoreCase("close")) {
System.exit(0);
}
else {
System.out.println("unknown command: ." + inputLine + ".");
}
}
}

public static void main(String[] args) {
//Next 3 lines Quelea specific
File nativeDir = new File("lib/native");
NativeLibrary.addSearchPath("libvlc", nativeDir.getAbsolutePath());
NativeLibrary.addSearchPath("vlc", nativeDir.getAbsolutePath());

PrintStream stream = null;
try {
stream = new PrintStream(new File(QueleaProperties.get().getQueleaUserHome(), "ooplog.txt"));
System.setErr(stream); //This is important, need to direct error stream somewhere
new OutOfProcessPlayer(Integer.parseInt(args[0]));
}
catch (Exception ex) {
ex.printStackTrace();
}
finally {
stream.close();
}
}
}


Hopefully, after you study the above two classes it should be pretty self-explanatory what's going on. The two classes talk to each other via the streams and react accordingly. The protocol there isn't complete - for now I've just completed what Quelea requires, though if I develop it into a library in its own right I'll complete the API obviously!

Conclusion


It's not trivial implementing out of process players in VLCJ, but neither is it ridiculously difficult and it does provide for trouble free playing if you get it right. The code above could definitely, and will almost definitely be improved - in terms of quality, completeness and error handling (there's nothing in there at the moment to cope with the external VM just disappearing or being closed externally, for instance.) And while communicating over streams like the above is reliable and doesn't need any big external libraries on top, it's not the fastest approach (more than good enough for my purpose however.)

However, as I already stated the purpose of this is to get the idea across, to provide some skeleton code for out of process VLCJ players, and to show that with a bit of work, it's entirely possible to get excellent video support in a Java application (even if it is done natively.)

I hope it proves useful to at least someone!

Java and Video - part 2

A while back I posted whining about the lack of decent video support in Java, and the fact that try as I might I just couldn't seem to find a decent one. I wrote this as part of my struggles trying to get video implemented properly in Quelea. The good news is that the trunk version of Quelea seems to have decent, cross platform and reliable video that's compatible with any format you throw at it. And it's all thanks to the VLCJ / VLC guys. The bad news is that, well, it's not as obvious or as easy as it might first look.

Before I go into details however, I think an honourable mention goes to Xuggler. It was the first package I really tried that seemed to be getting me somewhere - with really quite little effort after watching the tutorials I had a basic video application going that could play most file types. It couldn't do any seeking, pausing or the like but that I thought would be relatively easy. And, guess what - once again I was completely wrong. See, Xuggler isn't a high level video API. You don't point a video file and canvas at it and tell it to play one on the other. Instead, you have to open each packet in the file, check what stream it comes from, see if it's a video or an audio packet, deal with it appropriately and make sure the timestamps of the two match up (that's the really hard part)...

Initial results looked positive. But before long the video and audio on the videos drifted slightly out of sync. If there was a significant pause on the video because it was fetching it from disk or something like that, this problem got worse still. There were all sorts of corner cases as well that would've taken months to track down and cope with, both in terms of streaming, file formats, sync issues and so on. The guys on the discuss page were very friendly, and there were excellent tutorials provided - but as time wore on it became clearer and clearer Xuggler wasn't going to be for me. Beyond a certain point you were always going to be on your own, and I needed to go way beyond that point!

That's not to say Xuggler's useless, far from it. If you're doing any low level video stuff (transcoding, converting between formats, analysing frame / sound contents etc.) it's probably the best out there for Java. But that wasn't what I needed to do. The crux of the matter is, Xuggler just isn't really designed to be used in this way. It's not a high level video API, nor will it ever be.

At this point I really did think all my options were out. JMF was dead, no-one on the face of this earth can seem to get FMJ working / it's half dead too, VLC/VLCJ kept bringing the VM down, Xuggler was aimed at the wrong area for me, and to top it off my JavaFX2 hopes were dashed by finding out video-wise it only supported FLV, at least for now.

Rather than keep trying to find new things to use (I'd done a lot of Googling by now!) I decided to revisit my options and see if there was anything I'd missed. On the JMF and FMJ front this seemed very unlikely. The conclusions I came too seem to be reflected by many others for very valid reasons - there's not a lot you can say about projects that genuinely seem dead and don't work properly. And as for JavaFX, well it really does just support FLV and I really do need much more than that! I'd already taken a long, hard look at Xuggler; this left me with VLCJ and more specifically asking the following two questions:


  • Why did it seem to crash the VM every so often whenever I did anything complicated?

  • Could this be avoided?



This didn't take too long to find out: http://code.google.com/p/vlcj/wiki/Crashes
In fact, I'd already found and read the link before, but I'd brushed it aside because it basically seemed to say that there wasn't much hope with ever getting the thing to work 100% reliably. At least not without resorting to some out of process magic which was a route I really didn't want to go down.

After getting to this point though I did start to look down the out of process route, and it's this approach that I'm currently using in Quelea. In Windows it appears to work absolutely fine, I haven't managed to test anywhere else yet but in theory there's no reason why it shouldn't work on MacOS or Linux (potentially more native code might need to be executed on Mac to get it to work; this won't be an out of the box solution on Mac. But more on that later hopefully!)

To avoid creating one huge post, the code and more details about it will follow. But if you've been after a run down of how to do video bits in Java and come unstuck, I hoped this has helped.

In conclusion? If you want low level video manipulation, use Xuggler. If you want to play videos nicely in a variety of formats, use VLCJ with out of process players.