1080i MythTV MPEG-TS to stereo AVCHD
jamesp — Fri, 05/16/2008 - 15:05
Use:
This is useful for the rare 1080i MythTV recordings I've gotten via firewire from my cable box. For some reason, these recordings really confuse ProjectX and therefore can't be fixed. If you have any blips in the recording, then you will lose sync without ProjectX or the method below. The method below also does deinterlacing and samples sound down to stereo. The resultant file is playable on a PS3.
Chosen Tools:
- mencoder
- y4mscaler
- mjpeg tools
- x264 encoder
- HDTVtoMPEG2.exe running in wine
- MP4Box (GPAC)
- optionally screen
Recipe:
First, you need to cut commercials. HDTVtoMPEG2.exe is the only GUI program required, and usually for this I also run a lightweight VNC session (using fluxbox). The program is pretty self-explanatory, make sure your wine emulator can see the place your .ts file is, then "Add" it, scroll across the timeline and mark the commercials with include and exclude. Finally, make sure it will save an mpg file and process it. You don't want any demuxing as HDTVtoMPEG2 is a non-destructive cutter--it does not process any problems that are in your file.
I use screen to open four sessions. Three are used for the encode, while the fourth is so that I can do other stuff without messing up the command history. I also use three passes: two video and an audio pass.
Before firing off the encode, I use two pipes. One between mencoder and the mjpegtools, the second from the mjpegtools to x264. If you don't have them in your working directory, do
mkfifo stream.y4m
mkfifo stream-dint.y4m
In the first session, I use mencoder to render the video. Why mencoder you ask? Because you need the harddup filter to ensure that mplayer will keep things in sync. MPlayer does an excellent job rendering mpeg-ts files with problems, and keeping the sync. I must say here that av-sync is one of the most annoying things to me. I can deal with blocky artifacts, interlacing lines and jaggies, but I can't stand for the Kung-Fu dubbed movie look where the lips don't match the audio. On a side note, I am a big fan of Kung-Fu movies, and for those I will choose to watch in Chinese and read subtitles rather than the cheesy dubbing. But I digress. Another tip is you need to let mencoder see the whole stream each time so it can compensate for problems with harddup. Here's the mencoder command for the two video passes. Notice I basically run it twice with a sleep so that all the pipes are flushed before it tries again:
mencoder -ovc raw -of rawvideo -oac pcm -vf harddup,format=yuy2 -o stream.y4m <file cut using HDTVtoMPEG> && sleep 10 && mencoder -ovc raw -of rawvideo -oac pcm -vf harddup,format=yuy2 -o stream.y4m <file cut using HDTVtoMPEG>
In the second session, I use some YUV-domain tools to work with this video. The various YUV formats are uncompressed, so this is the reason you must use pipes, unless you want gigs and gigs of raw video in between. It actually would run slower, too, as there is no disk IO using pipes. For a hint on how big your file would be, look at the numbers in the brackets as mencoder is spitting out frames. Those are the bitrates in kilobits per second. Also, mencoder itself says next to the Trem the final file's size.
The file I was converting while writing this would've resulted in a 149GB video file for a 21 minute show. The command below will result in the interlaced YUY2-packed video being re-packed into a YUV4MPEG2 stream and then being deinterlaced for x264's consumption. yuvdeinterlace is a motion compensating filter, so should do well with fast-motion. As of this writing 1080p AVCHD is hard for linux computers to render because there is not yet support for rendering help from the video hardware that is present in some current video cards. Therefore for any single core systems, I would recommend shrinking this, or possibly using some other codec altogether. My goal was to get this into AVCHD, which could be authored to BluRay Discs or DVDs or streamed from my Media Cache to my PS3. To shrink, insert, as an example, a "-O size=960x540" in the y4mscaler part. It needs to be its own -O portion in addition to the chromass one there. There are many options to y4mscaler, "man y4mscaler" is your friend. Note that if you don't symmetrically change the resultion (ie halve both values), you may have to provide a sar value so that the aspect ratio is properly preserved.
cat stream.y4m | yuyvtoy4m -w 1920 -h 1080 -i t -r 30000:1001 | y4mscaler -O chromass=420jpeg | yuvdeinterlace > stream-dint.y4m
To down-rez to 720p:
cat stream.y4m | yuyvtoy4m -w 1920 -h 1080 -i t -r 30000:1001 | y4mscaler -O size=1280x720 -O chromass=420jpeg | yuvdeinterlace > stream-dint.y4m
For the third and final session I actually have a script I use that easily wraps up the options I've found that give tight AVCHD compression while remaining PS3 compatible.
--- cut here for x264-ps3.sh ---
#!/bin/bash
x264 -b 3 -m 6 -B $2 -p $3 -A "p8x8,b8x8,i4x4" -w --no-psnr --no-ssim -r 3 --mixed-refs --bime --b-rdo --level 41 -t 1 -o $1.mp4 --progress stream-dint.y4m
--- cut here for x264-ps3.sh ---
I must give credit for the options. I actually found them provided by an all-in-one mencoder script. I had to reverse engineer the mencoder-style options to figure out the x264 equivalent. The reason I didn't use that solution is that I had problems with using the mcdeint filter with mencoder on my Core 2 PCs, a CPU-specific problem I discovered online. I was unsatisfied with the non motion compensating filters. Cut and paste that script and save it if you don't have it. On some distros, the x264 you get doesn't have GPAC support and thus can't create .mp4 files. On those (Fedora is one I've dealt with for sure), change "$1.mp4" to "$1.mkv", as most of them can create matroska files. In either case, after saving the file, be sure to chmod a+rx x264-ps3.sh
it, then do this:
for pass in 1 2 ; do
./x264-ps3.hd <final video filename without extension> 9000 $pass
sleep 5
done
This encode is slow, no doubt about it, so you can really decide if 30-60% savings is worth the time. I have computers built to do this, so the time is not a question, so I'd just as well save the space. But if your only computer is going to be down for 9 hours 2-pass encoding some show, it may not be worth it to you.
The 9000 is the bitrate (in kbps), and seems to be a good trade-off in size and quality on my current batch of conversions, but this is highly dependent on the material, and thus, subject to change. This is why I made it an option. On some material, 9 mbps is not nearly enough for 1080p video, so adjust to taste. If you haven't started, you can scroll up to the mencoder command and create samples. You will add the "-ss" and "-endpos" options to pick a starting time and ending position (offset by the starting time). Use this to try to pick a portion of the video to encode. Keep in mind, though, that x264 will use that bitrate as an average to try to hit when doing a multipass encode. It will use bits more or less how it needs them to try to average out what bitrate you give it. So, a sample doesn't give it much room to work with, so it will end up being the bitrate you pick, where as when you do the full video, it can "steal" bits from the slow moving scenes to give more to the fast moving ones. Also, if you chose to scale down in session two, than you might be able to get away with a lower bitrate. I hate the vagueness I've gotten online, so I'll try to give numbers. With 1080p, I'd say the time to re-encode versus the space savings makes converting MPEG2 video to AVCHD not worth it if you need to go over 10mpbs, but that's my opinion. For my stuff, the 9000 gives about half the size, so I can fit an hour where I could only fit 30 minutes on my NAS. Since I have a couple boxes sitting around that could be doing this 24x7, it's worth it to me to put them to work saving space on the NAS. There will be another page detailing how to re-mux HDTV into a container the PS3 will play, should size not matter.
On another side note, I've found it easy to use gentoo for encoding boxes because you have fine control over the options built into your tools, there are overlays so you can easily live on the SVN-edge for mplayer/mencoder/ffmpeg. Also, all the Linux tools I'm using have ebuilds. On Ubuntu, I had to compile y4mscaler from source--the tarball I already had in /usr/portage/distfiles on another computer. On Fedora, I had to compile GPAC and then trick out x264 to use it (both using .spec files and rpm-build). But that is beyond this book. I am assuming some level of expertise with your distro of choice if you are going to be doing this stuff command-line. I don't advocate any distro over the others--I'm only mentioning my first hand experiences. I'm assuming you can google your way into, for example, finding HDTVtoMPEG2 and getting it put in your WINE installation. There is a ton of information out there, which is partially why I'm writing this. After spending lots of time using trial-and-error (and AVCHD encodes are S-L-O-W) and online searching, I've wanted to write out what I found that works for me. It could possibly work for you.
Once the hours or days go by, you will end up with a mp4 (or mkv) with AVC video. Now we need one more pass from mencoder to get the audio. I have confirmed this works. I'm not really sure that you need to do video, as I believe harddup modifies the video stream to match the audio. My final video was in-sync. Do this in session one:
mencoder -ovc xvid -xvidencopts bitrate=100 -vf scale=192:108,harddup -oac faac -faacopts quality=90:mpeg=2 -o <output filename>-audio.avi <file cut by HDTVtoMPEG>
This pass is to make sure mencoder sees the video and audio to take care of any problems. We don't really care about the video, which is why we are crunching it down to 192x108 and using a tiny bitrate. We will be throwing it away. This should move much faster than the video encodes. When it's done, we'll simply use MP4Box to repackage our stuff into an AVC:
MP4Box -fps 29.970 -aviraw audio <output filename>-audio.avi
mv <output filename>-audio_audio.raw <output filename>-audio.aac
MP4Box -fps 29.970 -add <output filename>-audio.aac -add <final video filename>.mp4 <final filename>.mp4
Make sure the final filename doesn't exist, otherwise MP4Box will add these two streams to the same file, which is probably not what you want. You may have to extract raw .h264 from the matroska video file if you had to use an inferior build of x264 without GPAC. Use mkvextract for that--googling or manpage reading for the details. You'd replace the output filename from that step instead of the second filename in the last MP4Box command above.
You should now have an AVCHD file, deinterlaced as well as it could be, and hopefully at the sweet spot of space / quality. At this point in my encodes, I would do a cp to get the file on my NAS, then make sure my MediaTomb server saw it, and watch it in on my PS3. You may go on to master it onto a BluRay on DVD. The advantage is you can fit about an hour of commercial-less TV in 1080p on a single layer Disc. I have no standalone BluRay player to test, but I've heard the PS3 is pretty much as picky about the format of the file so if it will play it, a stand alone BluRay should, too.