iPhone and iPod Touch compatible video encoding, server side.

Some days ago, I published a tutorial with some bash scripts to automate the whole process of encoding MP4 files with H264 video and HE-AAC audio which are playable by flash players and the like.

Since I’m working on a really big project which involves a “youtube like” behavior, i though it would be cool to also create iPhone / iPod compatible streams (I’m amazed at the fact that 3% of the people coming to this site is using iPod’s or iPhone’s)… so, I did some modifications to the original script for it to generate the compatible streams.

Now, how am i dealing with this… well… let me explain you:

The iPhone specification states:

Video formats supported: H.264 video, up to 1.5 Mbps, 640 by 480 pixels, 30 frames per second, Low-Complexity version of the H.264 Baseline Profile with AAC-LC audio up to 160 Kbps, 48kHz, stereo audio in .m4v, .mp4, and .mov file formats; H.264 video, up to 2.5 Mbps, 640 by 480 pixels, 30 frames per second, Baseline Profile up to Level 3.0 with AAC-LC audio up to 160 Kbps, 48kHz, stereo audio in .m4v, .mp4, and .mov file formats; MPEG-4 video, up to 2.5 Mbps, 640 by 480 pixels, 30 frames per second, Simple Profile with AAC-LC audio up to 160 Kbps, 48kHz, stereo audio in .m4v, .mp4, and .mov file formats

This means that the high profile streams that I am generating with the MKMP4 script won’t play, simple as that. So i have 2 possible approachs:

  1. I could just use a single low profile stream for the iPhone and iPod, and since the stream is H264 and low complexity AAC audio, it will just play fine on the flash players too.
  2. I could have 2 different files, one for the flash players (making use of b-frames, entropy coding, and a high profile stream, which substantially improves the final video quality).

I decided to use the option 2, mostly because i want to give desktop users as much quality as possible, and i want to use the HE-AAC encoder to save some extra bandwidth that i can instead use for the video stream, so I am using 2 different files, one for desktop users, and a different one for iPhone or iPod users.

To check if the user is visiting the site with a desktop browser or a Mobile Safari version is really simple: Let’s consider a PHP based scenario.

The first thing that i had to do is to create a simple function to check the browser’s user-agent string.

function isIphone() {
$agent = $_SERVER['HTTP_USER_AGENT'];
if (stristr($agent, 'iPhone') == TRUE || stristr($agent, 'iPod') == TRUE) {
echo '<!--iPhone or iPod Touch detected-->;';
return TRUE;
} else {
echo '<!--iPhone or iPod Touch not detected-->';
return FALSE;
}
}

This function is self explanatory for anybody with basic PHP skills, basically it checks the user-agent string and returns TRUE if it contains the words “iPhone” or “iPod”.

So, then in your template you could make something like this:

if (!isIphone()) {
// Code to display the standard high profile stream goes here.
} else {
// Code to display the iPhone compatible stream goes here.
}

Pretty simple.

The updated MKMP4 script.

This is an updated version of my MKMP4 encoding script, basically, it is almost the same thing than the original one published here, with the addition of an extra -i switch that you can use if you want to create an iPhone compatible stream.

Important: FAAC encoder is a requeriment for this!

This version of the MKMP4 script uses the FAAC encoder that you can download by going here. Be sure that you have correctly compiled and installed it BEFORE trying to use this script on your desktop or server.

#! /bin/bash

# This bash script has been created by Diego Massanti
# www.massanti.com
#
# set -x # uncomment this for debugging
shopt -s nullglob # so the glob expands to nothing when there are no .mov files.

# User settings:

fileext=".mov" # Set this variable to the video file extension that you want to search for when encoding whole directories.
onlythumbs=FALSE # IF you set this to TRUE, the script will only generate thumbnails.
# End user settings
# You should not change anything below this line unless you know what you are doing.

usage()
{
cat << EOF
usage: $0 -f  [-w ] -b  [-q][-i][-k][-t][-o ]

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
* h264 video with he-aac audio encoding script by Diego Massanti. *
*                 October 2008,  Made in Argentina.               *
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

OPTIONS:
 -h    Show this message
 -f    Path to the file to encode < REQUIRED
 -w    Resize video to fit inside this width while keeping the aspect ratio < OPTIONAL
 -b    Desired video bitrate < REQUIRED
 -o    Output directory for compressed media < OPTIONAL (will use same as source if ommited)
 -q     Better quality encoding using 2 passes (slower) < OPTIONAL
 -k    Do not delete (keep) temporary files < OPTIONAL
EOF
}
audiobitrate=48000
platform=""
uname=`uname`
if [ $uname == "Darwin" ]; then
 ## 99% of chances that this is a Mac
 platform="Mac"
else
 platform="Linux"
fi

width=""
bitrate=
deltemp=TRUE
quality=FALSE
iphone=FALSE
poster=FALSE
filename=""
rsize=""
outdir=`pwd`
outFileName=""
itemVideoBitrate=""

while getopts ":f:w:o:b:qktvi" OPTION; do
 case $OPTION in
 w ) width=$OPTARG;;
 f ) filename=$OPTARG;;
 b ) bitrate=$OPTARG;;
 k ) deltemp=FALSE;;
 q ) quality=TRUE;;
 t ) poster=TRUE;;
 i ) iphone=TRUE;;
 o ) outdir=`cd "$OPTARG"; pwd`;;
 h ) usage;;
 \? ) usage
 exit 1;;
 * ) echo $usage
 exit 1;;
 esac
done

if [ "$1" == "" ]; then
 usage
 exit 0
fi

if [ "$filename" == "" ]; then
 echo
 echo you MUST supply a file to encode!, use the -f parameter. i.e: -f mymovie.avi
 echo
 usage
 exit 1
fi
if [ "$bitrate" == "" ]; then
 echo
 echo you MUST specify a target bitrate!, use the -b parameter. i.e: -b 512
 echo
 usage
 exit 1
fi

if [ "$width" != "" ]; then
 rsize="-vf scale=$width:-3"
 rsizemsg="fit into $width pixels wide"
else
 rsize=""
 rsizemsg="Movie is not being resized."
fi

# Check if output directory exists
if [ -d "$outdir" ]; then
 echo "Target directory set to: $outdir"

else
 echo "Error: '$outdir' does not exist!!"
 echo "Can not continue, please make sure that the specified output directory exists. Exiting..."
 exit 1
fi

# Start the magic...

showMessage() {
 echo "MKMP4 : "$1""
}

initialize() {
 if [ -d "$filename" ]; then
 $filename = `cd $filename; pwd`
 showMessage "Output directory set to: $outdir."
 showMessage "Triggering the encoder with all media files in directory: $filename."
 for file in "$filename"*$fileext; do
 if [ -f "$file" ]; then
 showMessage "Will encode file: $file"
 startEncoders "$file"

 fi
 done
 elif [ -f "$filename" ]; then
 showMessage "Triggering the encoder with source media: $filename."
 startEncoders "$filename"

 else
 # input is no file and no directory ? WTF ?
 echo "Something really weird has happened here: $filename is not valid, exiting..."
 exit 1
 fi
}

startEncoders() {
 if [ $onlythumbs == "FALSE" ]; then
 firstPass "$1"
 if [ "$quality" == "TRUE" ]; then
 secondPass "$1"
 fi
 extractAudio "$1"
 encodeAAC "$1"
 mux "$1"
 fi
 if [ $poster == "TRUE" ]; then
 createThumbnail "$1"
 fi
 if [ $deltemp == "TRUE" ]; then
 cleanTemporaryFiles "$1"
 fi
}
firstPass() {
 local iFile="${1##*/}"    # remove directory and keep only file name.
 # First, lets get some basical information about this file, as FPS.
 MOVIE_FPS=`midentify "$1" | grep FPS | cut -d = -f 2`
 MOVIE_LENGTH = `midentify "$1" | grep "ID_LENGTH" | cut -d = -f 2 | cut -d . -f 1`
 let "THUMB_SECONDS = $MOVIE_LENGTH / 2"
 # Lets print some info to stdout.
 showMessage "Encoding: $1"
 showMessage "Resizing to: $rsizemsg."
 showMessage "Total Bitrate: $bitrate kbps."
 let "caudiobitrate = $audiobitrate / 1000"
 let "itemVideoBitrate = $bitrate - $caudiobitrate"
 showMessage "Video Bitrate: $itemVideoBitrate kbps."
 showMessage "Audio Bitrate: $caudiobitrate kbps."
 showMessage "Platform: $platform."
 # Lets start the encoding for the first pass.
 showMessage "Starting video encoding pass 1..."
 if [ $iphone == "TRUE" ]; then
 mencoder "$1" -o "$outdir/${iFile%%.*}_temp.264" -demuxer lavf -passlogfile "$outdir/${iFile%%.*}"_temp.log $rsize -ssf lgb=0.2 -ovc x264 -x264encopts bitrate=$itemVideoBitrate:frameref=1:nocabac:bframes=0:b_adapt:b_pyramid:weight_b:partitions=all:me=umh:subq=6:trellis=2:brdo:threads=auto:pass=1:analyse=all -of rawvideo -nosound
 else
 mencoder "$1" -o "$outdir/${iFile%%.*}_temp.264" -demuxer lavf -passlogfile "$outdir/${iFile%%.*}"_temp.log $rsize -ssf lgb=0.2 -ovc x264 -x264encopts bitrate=$itemVideoBitrate:frameref=8:bframes=3:b_adapt:b_pyramid:weight_b:partitions=all:8x8dct:me=umh:subq=6:trellis=2:brdo:threads=auto:pass=1:analyse=all -of rawvideo -nosound
 fi
}

secondPass() {
 local iFile="${1##*/}"    # remove directory and keep only file name.
 showMessage "Starting video encoding pass 2..."
 if [ $iphone == "TRUE" ]; then
 mencoder "$1" -o "$outdir/${iFile%%.*}_temp.264" -demuxer lavf -passlogfile "$outdir/${iFile%%.*}"_temp.log $rsize -ssf lgb=0.2 -ovc x264 -x264encopts bitrate=$itemVideoBitrate:frameref=1:nocabac:bframes=0:b_adapt:b_pyramid:weight_b:partitions=all:me=umh:subq=6:trellis=2:brdo:threads=auto:pass=2:analyse=all -of rawvideo -nosound
 else
 mencoder "$1" -o "$outdir/${iFile%%.*}_temp.264" -demuxer lavf -passlogfile "$outdir/${iFile%%.*}"_temp.log $rsize -ssf lgb=0.2 -ovc x264 -x264encopts bitrate=$itemVideoBitrate:frameref=8:bframes=3:b_adapt:b_pyramid:weight_b:partitions=all:8x8dct:me=umh:subq=6:trellis=2:brdo:threads=auto:pass=2:analyse=all -of rawvideo -nosound
 fi
}

extractAudio() {
 local iFile="${1##*/}"    # remove directory and keep only file name.
 showMessage "Extracting Audio..."
 if [ $iphone == "TRUE" ]; then
 mplayer "$1" -af resample=48000:0:2,channels=2,volnorm=1:0.25,format=s16le -ao pcm:file="$outdir/${iFile%.*}_temp.wav" -vc dummy -vo null
 else
 mplayer "$1" -af resample=48000:0:2,channels=2,volnorm=1:0.25,format=s16le -ao pcm:file="$outdir/${iFile%.*}_temp.wav" -vc dummy -vo null
 fi

}

encodeAAC() {
 local iFile="${1##*/}"    # remove directory and keep only file name.
 if [ $iphone == "TRUE" ]; then
 faac "$outdir/${iFile%.*}_temp.wav" -q 100 -o "$outdir/${iFile%.*}_temp.mp4"
 else
 if [ "$platform" == "Mac" ]; then
 enhAacPlusEnc "$outdir/${iFile%.*}_temp.wav" "$outdir/${iFile%.*}_temp.aac" $audiobitrate s
 else
 neroAacEnc -br 48000 -he -if $outdir/${iFile%.*}_temp.wav -of $outdir/${iFile%.*}_temp.mp4
 fi
 fi
}

mux() {
 local iFile="${1##*/}"    # remove directory and keep only file name.
 MP4Box -add "$outdir/${iFile%.*}_temp.264#video:fps=$MOVIE_FPS" "$outdir/${iFile%.*}.m4v"
 if [ $iphone == "TRUE" ]; then
 MP4Box -add "$outdir/${iFile%.*}_temp.mp4#audio" "$outdir/${iFile%.*}.m4v"
 else
 if [ "$platform" == "Mac" ]; then
 MP4Box -add "$outdir/${iFile%.*}_temp.aac" -sbr "$outdir/${iFile%.*}.m4v"
 else
 MP4Box -add "$outdir/${iFile%.*}_temp.mp4#audio" "$outdir/${iFile%.*}.m4v"
 fi
 fi

 name=${iFile%.*}
 album="The Mac Video Archive"
 author="Apple Computer // massanti.com"
 comment="Professionally encoded by Diego Massanti"
 created="2008"
 MP4Box -inter 500 -itags album="$album":artist="$author":comment="$comment":created="$created":name="$name" -lang English "$outdir/${iFile%.*}".m4v
}

createThumbnail() {
 local iFile="${1##*/}"    # remove directory and keep only file name.
 mplayer "$1" -ss 10 -nosound $rsize -sws 7 -vo jpeg:outdir=$outdir -frames 1
 mv "$outdir/00000001.jpg" "$outdir/${iFile%.*}.jpg"
 mplayer "$1" -ss 10 -nosound -vf scale=150:-3 -sws 7 -vo jpeg:outdir=$outdir -frames 1
 mv "$outdir/00000001.jpg" "$outdir/${iFile%.*}_small.jpg"
}

cleanTemporaryFiles() {
 local iFile="${1##*/}"    # remove directory and keep only file name.
 showMessage "Removing temporary files..."
 rm "$outdir/${iFile%.*}"_temp*
}

initialize

The following example command line will encode the video stream in my_video.avi as a low profile H264 without b-frames, will encode the audio track as a Low Complexity AAC stream and will mux everything inside a M4V file which will play on the iPhone, iPod Touch, Apple TV and Quicktime / iTunes.

./mkmp4 -f my_video.avi -b 512 -w 640 -q -t -i

For a detailed explanation of all the supported command line switches, you can visit the original MKMP4 post.

Some final considerations:

  • Due to the fact that I am forced to use a low complexity AAC encoder for the iPhone streams, the audio bandwidth is hardcoded at around 100kbps inside the script, that would give you an audio quality similar to the one you get with the HE-AAC encoder at 48kbps, feel free to experiment with different values for this.
  • Using a low profile H264 stream gives me iPhone compatibility, but at the expense of losing some video quality, clearly, a 512kbps stream looks much better when we use b-frames and entropy coding so unless you really care about iPhone and iPod users like I do, you should use high profile streams.
  • If you are visiting this page with an iPhone or iPod and want to see how all this works, feel free to visit this section of the Apple Video Archive here to see the magic happening: If you visit the page with a desktop browser, it will simply show the standard flash player, BUT when it detects you are on a Mobile Safari platform, it will automatically show you the iPhone compatible stream (click here to see how this all looks) and properly embed the quicktime player on the page.

22 Responses to “iPhone and iPod Touch compatible video encoding, server side.”

  1. Amazing.

    For a third time you help me in something I was just incidentally searching. Awesome. Thanks a bunch.

    by eduo, October 13th, 2008 at 12:15 pm


  2. LOL, im glad that you find all this so useful, care to tell me where should i send the bill ? :p
    cheers :)

    by Diego Massanti, October 13th, 2008 at 12:18 pm


  3. I was today using this script and was wondering… If it even worth using midentify to check if the video track can be used for the iTouch/iPhone?

    A downside of reencoding is possible loss of quality and long encoding times (how long depends on the machine). I was wondering if it would make sense to add a parameter where, if the target file already has a video track compatible with the mp4 iPhone specifications (for example, DiVX within bitrates and resolutions), to demux and remux that instead of re-coding.

    by eduo, October 30th, 2008 at 6:09 pm


  4. Re-muxing without re-encoding is of course possible, but…. is DiVX compatible with the iPhone ? i wasn’t aware of that to be honest.
    Only thing you need in order to remux is MP4Box.

    by Diego Massanti, November 1st, 2008 at 8:22 pm


  5. DivX is compatible with the iPhone. Encoding is *MUCH* faster but final size is usually bigger than H.264.

    This is a screenshot of a file being played on the iPod: http://www.flickr.com/photos/eduo/2994952697/

    This is the midentify.sh output for it:

    ID_VIDEO_ID=0
    ID_AUDIO_ID=1
    ID_FILENAME=/Volumes/Malfoy/iTunes/iTunes\ Music/TV\ Shows/Seinfeld/The\ Note.m4v
    ID_DEMUXER=lavfpref
    ID_VIDEO_FORMAT=mp4v
    ID_VIDEO_BITRATE=0
    ID_VIDEO_WIDTH=544
    ID_VIDEO_HEIGHT=416
    ID_VIDEO_FPS=25.000
    ID_VIDEO_ASPECT=1.3077
    ID_AUDIO_FORMAT=255
    ID_AUDIO_BITRATE=0
    ID_AUDIO_RATE=44100
    ID_AUDIO_NCH=2
    ID_LENGTH=1345.16
    ID_SEEKABLE=1
    ID_CHAPTERS=0
    ID_VIDEO_CODEC=ffodivx
    ID_AUDIO_BITRATE=128000
    ID_AUDIO_RATE=44100
    ID_AUDIO_NCH=2
    ID_AUDIO_CODEC=faad

    Quickly scanning my ipod library shows I have around 50/50 for ffodivx and ffh264 (all audio is faad).

    I don’t know about the AppleTV, tho’. I don’t have one. I use XBMC/Plex for watching video on TV. I believe every time “MP4 Video” is being mentioned it’s assumed divx is supported but I could be reading too much into it.

    I generally prefer H264, as it makes smaller files and, if time is not an issue, the result is worth it. But an option to save CPU and time could benefit people using the script. Especially using it massively.

    Saw your comment in the blog. I obviously agree with what you said, as is clear by the post itself :D

    by eduo, November 2nd, 2008 at 12:04 pm


  6. hmmmmmmmm this is interesting… the iPhone/iTouch is NOT supposed to be able to play that file… nevertheless it is working for you…
    Is your iTouch jailbroken ? are you using any third party player or something strange ?

    by Diego Massanti, November 2nd, 2008 at 12:09 pm


  7. No. It’s all stock. I’ve been using this command since firmware 1.x: http://pastebin.com/f3ace28f9

    As you can I only specify “mp4” as the container and “mpeg4” as the codec. Before that I saw the specs for the iTouch (pasted from a document I keep here with most of what I “researched” back then):

    Audio
    Frequency response: 20Hz to 20,000Hz
    Audio formats supported: AAC, Protected AAC, MP3, MP3 VBR, Audible (formats 1, 2, and 3), Apple Lossless, AIFF, and WAV

    Video
    Video formats supported:
    H.264 up to 1.5 Mbps, 640 by 480, 30 fps, Low-Complexity version of the H.264 Baseline Profile
    H.264 up to 768 Kbps, 320 by 240, 30 fps, Baseline Profile up to Level 1.3
    MPEG-4 up to 2.5 Mbps, 640 by 480, 30 fps, Simple Profile

    AAC-LC audio up to 160 Kbps, 48kHz, stereo audio in .m4v, .mp4, and .mov file formats.

    As you see it specifies it uses MPEG-4 as the video codec. I scratched a little more (you can see how Apple doesn’t specify what “variant” or MPEG-4 Video it is supposed to be, so I assumed it’d be the ISO Standard):

    http://www.apple.com/quicktime/technologies/mpeg4/

    Apple isn’t very clear there, so after digging around I realized that DivX was the basis for the default MPEG-4 video codec (even if a multitude of codecs can be in an MP4) in the same way Quicktime 7 made the basis for the container itself.

    I actually copied the video file with TuneAid to make sure iTunes wasn’t converting the video or anything.

    So the limits outlined there for the video codec where it says MPEG-4 apply to DivX, really.

    by eduo, November 2nd, 2008 at 1:58 pm


  8. Hmm. Linefeeds don’t translate very well when the comment is submitted, do they? :)

    In case you want to IM or Twitter my contacts are all in my contact page.

    by eduo, November 2nd, 2008 at 2:00 pm


  9. Ok, i think we just discovered something unknown then ?… i mean, i don’t recall reading about the iPhone being able to play this type of media anywhere before…
    I’m with a friend who owns an iPhone and I will ask him to perform some tests in the meantime :D

    by Diego Massanti, November 2nd, 2008 at 2:05 pm


  10. If you want I can provide a sample file. No problem. I should be able to come up with a small one.

    by eduo, November 2nd, 2008 at 2:16 pm


  11. Indeed, i would like to take a look at one of those files.

    by Diego Massanti, November 2nd, 2008 at 2:32 pm


  12. Samples here:
    ffh264 file – http://eduo.info/hdp/BL1.m4v
    ffodivx file – http://eduo.info/hdp/BL2.mp4

    Both play perfectly on the iPod Touch. The extensions don’t mean anything. I change them to keep track of the files I’ve already tagged (some tags and especially styled subtitles won’t show on the iPod unless the track is .m4v)

    by eduo, November 2nd, 2008 at 7:21 pm


  13. Hey Diego,

    Fantastic script, I’ve just been using the x264 one which is brilliant!

    I’ve got an iPhone 3g with 2.1 firmware and thought this would be a great way to convert files to play on my iPhone.

    However I tried some clips from your apple video archive and when trying to play I get”This movie format is not supported” on my iPhone.

    Its not jailbroken so I find it strange I get that error :-(

    Eudo:

    Your ffh264 and ffodivx both play fine on my iPhone (for reference) which is extremely interesting.

    Tristan.

    by Tristan, November 7th, 2008 at 8:19 am


  14. Hey Tristan, thanks for the comments. Regarding your problem with the videos on this site, i had not the time to re-encode all of them yet, only the “iPhone” and “Get a Mac” sections are “iPhone/iPod Compatible”, try any of those sections and report back if you have the time.
    Thanks.

    by Diego Massanti, November 7th, 2008 at 10:26 am


  15. Hey Diego,

    Just tried “iPhone Teaser: Hello” and that worked perfectly, strangely the m4v I made using the script still shows the “This movie format is not supported” error which is a shame but at least its my problem, not the scripts.

    Thanks for some great tips!

    Tris.

    by Tristan, November 7th, 2008 at 10:32 am


  16. Found the issue…

    In the above scripts you’re missing “level_idc=30” in the mencoder passes for the iPhone/iPod, this seems to resolve the issue of “This movie format is not supported” for me.

    Hope this helps others, loving the scripts!!

    Tris.

    by Tristan, November 7th, 2008 at 11:15 am


  17. Hi Diego,

    After some more encoding I’ve come across another issue which I’ve not yet found the fix…

    It seems when encoding some videos I’m getting dropped frames, I tried to fix this by adding -vf harddup but it still happens. As mencoder processes the video with no audio it unfortunately results in sync issues.

    Do you (or anyone else) have any ideas which may solve this?

    Many thanks,

    Tristan.

    by Tristan, November 14th, 2008 at 3:57 pm


  18. I have discovered that I can encode 720X480 24 fps video for the iphone 3G and it works great. I’m using MPEG Streamclip. I then use YAMB to set the widescreen flag and mux the video and audio. You must use a custom 32/27 aspect ratio setting to get perfect 853X480 output. I tried this because I noticed that “digital copy” movies are encoded at 720X480 and play perfectly in the iphone 3G.

    by Scott, December 1st, 2008 at 10:15 pm


  19. Hi, I tried your apple ad page for iPhone (with latest fw) and the iPhone can’t play your videos.
    I tried with an iTouch as weel (older fw) and it can only play the sound.
    I’m searching a solution to use encoded video in my website like you

    by raskar, December 15th, 2008 at 9:03 pm


  20. Yes, im aware of that, interestingly enough, it looks like something changed when it comes to video playing in the latest iPhone/Ipod’s… im investigating whats the issue.

    by Diego Massanti, March 10th, 2009 at 2:40 am


  21. Thanks for this. Had to add ‘no8x8dct’ to the x264encopt in order to make the H.264 stream baseline compatible. Also, it doesn’t hurt to add ‘level_idc=30’ to force the H.264 level to 3.0. I think the latest builds of x264 figure out the required level and profile, which is why video stream was initially High profile (8×8 DCT on by default=High profile).

    by Ron, August 31st, 2009 at 4:44 pm


  22. This does not work in latest MEncoder SVN-r32021-4.1.2 2010 version, any 1 has success make this work on latest mencoder?

    by boon, August 27th, 2010 at 6:30 am