How best to rip/compress Blu-ray to MKV etc.? [Archive]

BLKMGK

8th March 2008, 23:51

Okay, HD-DVD proved to be fairly straightforward (http://forum.doom9.org/showthread.php?t=135361) once I talked to enough people and got some help. I've got my entire collection ripped and compressed to a server and they play terrific! However HD-DVD is pretty much toast now and I'm not even finding any good sales.

So, I have purchased a 4x BluRay drive. I figured surely this wouldn't be SO much different, right? Well, not so much! I've made SOME progress after reading this thread (http://forum.slysoft.com/showthread.php?t=11770). However the focus of that thread appears to be different, thus my posting here.

I have managed to identify the two files that made up my movie (The Patriot). I have joined them together with TSsplitter. I have remuxed the the result with TsRemux such that I have just one audio file and one video file - full resolution. These are in a .M2TS format. While XBMC will probably play that it's 32+Gigs and a bit "bulky". My HD-DVD weigh in at about 10-13Gigs which is more like it - I'm not looking to burn to media.

With an HD-DVD I'd split out the audio and video using eac3to, use the calculated frames in an AVIsynth script blah blah, compress the video with meGUI using x264, mux them back with MKVmerge, and call it a day. Unfortunately I cannot find (yet) anything nearly as helpful as eac3to for m2ts files although it looks like some audio conversion and channel flipping can be done with that tool - somehow.:confused:

So, how do I get to my end goal from here? I considered building a graph with Haali and FFMPEG with just the video and trying to compress it but I've not got the frames calculated for a script so I worry about synch. I also have no idea how to pull out the audio:bang: I'm stuck!

I'm NO pro at this but with help I've managed HD-DVD. Surely others are looking to do the same thing, has anyone come up with a work flow that consistently works? Surround sound, x264 compression, and reasonable file size with no noticable degradation in visual quality is what I'm after. Using PDVD isn't an option as Linux and XBMC is my target platform.:(

P.S. I know better than to buy anything FOX not to worry! A shame too, Independance Day would've rocked:mad:

GeeForce11

9th March 2008, 09:46

I've encoded quite a few BD discs to AVI like this:
http://forum.videohelp.com/topic346837.html

You could try this one:
http://forum.videohelp.com/topic347525.html
http://forum.videohelp.com/topic333634.html

I've found TSMuxer to work better than TSReMux or TSSplitter with the m2ts files in case of joining/extracting tracks.

I've extracted the master DTS-HD audio from m2ts with xport, and then encoded to AC3 with EAC3to.

BLKMGK

9th March 2008, 15:41

Thank you for the response! The last method you mentioned is actually sort of how I was considering doing this - or trying anyway. So far I've been trying to follow along with this http://www.avsforum.com/avs-vb/showthread.php?p=13318652#post13318652 and that has helped but it's slow going. I have the two m2ts files I need, have joined them, and have stripped all but the single audio and video track out. Using xport I've split those two apart and am now trying to use eac3to to put the video into a MKV container to compress as I would an HD-DVD. eac3to is hanging though partway through - doh! I've yet to touch the audio.:bang:

What you've posted should help - my thanks! All in all this seems WAY harder than HD-DVD dammit. I am worried a great deal about audio synch and maintaining surround sound, have you found synch to be an issue? I'm hoping I can puzzle out a consistent workpath and not hit too many dead ends. I do not want to have to become a full on expert at this just to be able to watch my movies:mad:

I appreciate the pointers, those were docs I'd not seen previously. If I can come up with something that layman can follow that's any different than what others have done I'll write it up. fingers crossed.... <sigh>

Edit: Madshi, author of eac3to, says he will be upgrading (http://forum.doom9.org/showthread.php?p=1110475#post1110475) his tool in the coming weeks to handle m2ts format files. That will be huge IMO.

GeeForce11

9th March 2008, 19:26

From what I noticed from the different BD discs I encoded, that sometimes one encoding process will not work properly with one disc, but it does with the other. :bang: The way I see it so far everything is trial and error.

My goal is a bit different, I made XviD/AC3 encoded AVI backups of the movies with one click encoding as described in the first link of the previous post. The encoding(total work) per movie takes 8-10hrs on my 2.7GHz AMD X2. The backups are 4-5GB each, 6.4Mbit/s XviD @ 1920x1080 and 640kbit/s AC3 5.1.

Luck with your project :)

BLKMGK

9th March 2008, 21:11

Well, am still fooling around with this one disk. I've manged to rip it again using TSmuxeR which leaves me with an AC3 and a .264 file. What are you doing at that point? One click encoding in meGUI is laughing at me and VLC won't play the file. When you have multiple .m2ts files what are you doing with them exactly and what does the output look like? I'm going to have to read over those posts carefully, this is driving me nutz!

Heh, the transcoding of just one of the Planet Earth HD-DVDs I just did took 19 hours and shaved 10Gigs off the original file - down to 11Gigs. That's on an overclocked C2D E8400 @3.8Ghz.:rock: My old AMD was taking as much as 80 hours to process some of the early files I did:disagree: Processing this stuff is NO joke and will seriously test the stability and cooling of a system. I can play games all day at 4Ghz but cannot always process a full video at that speed and there's nothing like throwing away 10hours of work to make you back a clock down to get through it next time.

GeeForce11

9th March 2008, 23:03

I didn't have a case of multiple m2ts files yet, but this is what I would do:
- BDEdit would list the order of the files the whole movie is put together
- I would join all these m2ts files accordingly with TSMuxer to get the whole movie in one m2ts file for encoding.
- for example with Jake Ludington (http://www.topdrawerdownloads.com/download/104913) mod of VDubMod is one click encoding of the m2ts to avi or mkv. I encode the video with XviD and direct stream copy of the master audio track. XviD seems like 3x faster than H264 encoding, one of the reasons I went with it. I could of kept the original uncompressed DTS-HD master track (approx 3GB), but to further reduce the size of the avi file for storage I encoded the audio to AC3. It happened once that for some reason the master track was not processed like this, but the next audio track was processed automatically, and I had to demux the master audio track from the m2ts with xport and encode with EAC3to to AC3 and then mux it with the XviD video.

It looks that if you don't want avi, but mp4, video and audio needs to be muxed out of the m2ts and encoded separately and then muxed back. This seemed for me more complicated and with more steps, another reason I went with avi.

BLKMGK

9th March 2008, 23:27

Okay, tomorrow evening I will checkout the VDubMod code for sure along with the mods. AC3 is fine by me and it does save a ton of space! I'm shooting for a MKV container, 5.1 soundtrack, and high bitrate H.264 via x264. That's what I've done with HD-DVD and it has worked out perfectly. If I can do the same with BD than I'll be very happy.

Thanks!

BLKMGK

10th March 2008, 20:03

Okay, Tsmux'd the file to my drive, loaded up the virtualDubMod code, and boom - stuck! VirtualDubMod doesn't accept M2TS format files. Are you doing something to them prior to bringing them in?:bang:

GeeForce11

11th March 2008, 10:56

I just had the Combined Community Codec Pack (CCCP) with haali splitter and AviSynth installed, and the modded VDubMod imports the m2ts like this:
http://images.jakeludington.com/vdm/vdmopen_m2ts.png
Make sure to select on the bottom AviSynth Template: "m2ts editing".

You won't need to create .avs AviSynth file to be able to import the m2ts, it will be created automatically first time you import the file. Once the .avs is created I also imported the m2ts.avs straight to MEGui, but didn't do the encoding/conversion there, but in VDubMod.

BLKMGK

11th March 2008, 16:28

Ah! Now we're cooking with gas - I have a job queued up now :bowdown: I'm a bit confused on how you took this into meGUI though, did you simply take the template from this program and use it with meGUI or did you do something that would have to be done for each video converted? I see the m2ts profile in avst format but am not familiar with that.

As soon as my current 1st pass job in meGUI finishes up I'll fire this one off and see hwo it goes. I took the x264 commandline from meGUI and just pasted it into the x264 config for this one so it ought to be interesting to see how it processes:D

Thanks!

P.S. Hrm, looks like I now have access to all sorts of filters too! Not sure I want to mess with those too much just yet but that could prove interesting when I have some time - maybe try them on a trailer.

GeeForce11

11th March 2008, 17:07

You don't have to do anything, just have the haali splitter and AVISynth installed already before you try to open the m2ts with VDubMod(the version i posted the link for in the first post has all the mods and templates included). It opens up the m2ts and creates the .avs file automatically, what you can import straight to MEGui. Once you open the m2ts in this VDubMod you can do the editing/converting, or close it and continue to work with the m2ts in MEGui.

http://www.jakeludington.com/camcorder/20070118_how_to_edit_avchd_m2ts_files_from_sony_hd rsr1_camcorders.html

BLKMGK

12th March 2008, 09:16

Ah okay, I hadn't noticed the AVI script created when I opened the file. What's odd is what's IN that script - it simply has an entry in it about using DirectShow, I'll post it when I'm at home. I'm FAR from an expert with AVIsynth scripts but this seems pretty simplistic! What I was using with HD-DVD was even a little more complicated than this <shrug>

Last night I ran the 32ig file I've got through VDubMod and the result was a 5Gig file - this with max bitrate - that completed with "warnings" but I've yet to find what those were. My attempt to use the commandline generated by the meGUI settings failed, I think the x264 program versions are out of synch just for starters. In any case I've got this setup to run through meGUI including the audio track so we'll see what the two look like when it's all done. I've got a job ahead of it though that looks like it will take an entire day so it will be a bit before I've got anything to look at :o When done I ought to have a MKV video and an AC3 track to mux together with MKVmerge, audio synch is a worry so we'll see how it goes.

Thank you for further clarifying!:bowdown:

GeeForce11

12th March 2008, 11:54

I know that avs thing is as simple as it could be, you could create that file yourself.

Since you have the video already from VDubMod, I'd extract the wanted audio track from the m2ts with xport and encode it with EAC3to to AC3. This should take a few minutes. Then muxing the audio and video also should take a few minutes, and you're done. :)

BLKMGK

12th March 2008, 19:32

Well, the compressed video from VDubMod fails to play correctly in XBMC. I'll see how it goes when the same video comes out of meGUI - fingers crossed! At least I feel like I'm making some progress, will know sometime late tomorrow maybe.:rock:

GeeForce11

12th March 2008, 22:47

I would be curious how would a third XviD compression AVI movie made with VDubMod compare quality wise to the MP4 you make.

BLKMGK

13th March 2008, 19:05

That might be interesting to try - particularly if it doesn't take 34hours to compress:bang: I admit I'm using some settings that are a bit nutz but HD-DVD weren't nearly so objectionable time wise. I'll explore using XVID when this is done and the two HD-DVD following it that are stacked in the queue.

GeeForce11

14th March 2008, 10:17

I would experiment on a few minute clip instead of working with a whole movie.
What speed do you get encoding to H264? 2-3fps?

BLKMGK

14th March 2008, 18:10

I would experiment on a few minute clip instead of working with a whole movie.
What speed do you get encoding to H264? 2-3fps?

Aww, where's the fun in a few minute clip?:p I'll see what kind of encoding rate I get from it - if we're talking a few hours no biggie but if it's going to take nearly as long then I'll be forced to use shorter pieces.

Currently using x264 I'm seeing encodes of just about 1.9 frames per second:bang: That's on an E8400 C2D cranked to 3.8Ghz. I have to build a system to replace my slower AMD box which has just puked badly, I'm pondering a Q6600 or another E8400 so that I can offload some of this to it. I do not think I can wait any longer for the next 45nm quads coming out nor am I sure they would buy me anything over an overclocked Q6600. <sigh>

GeeForce11

16th March 2008, 10:21

BLKMGK

16th March 2008, 10:55

From what I see during encoding, these programs don't use multiple cores, CPU usage is always 50-60% on a dual core machine, so there won't be any benefits on the Quad core, at least for the time being until multi threaded software comes out. :(

Umm which programs? X264 most certainly DOES use multiple cores on my machine and in fact on playback the H.264 decoder on Linux XBMC uses multiple cores too. As it happens when I was trying to decode audio for this BD media it died, I aborted and kicked off the video transcoding. I just noticed while doing another HD-DVD that framerate was slow. Looking I found one core consumed by a zombied program from that audio experience. Killed it and performance is back. On my machine at least, using meGUI, I'm seeing improvements with multiple cores. A quad, even at slower clock, ought to be able to beat my high clocked dual for work like this. Am making a purchase decision today or tomorrow, wish I knew when the new 45nm Intels were going to hit the street :bang: I do not, yet, know if XVID will use multiple cores. If it doesn't I'll kick off a second job:D

GeeForce11

16th March 2008, 11:20

I encoded with XviD in VDubMod and used only one core. :( I'll check if meGUI and X264 uses both cores. :o

BLKMGK

16th March 2008, 12:16

I encoded with XviD in VDubMod and used only one core. :( I'll check if meGUI and X264 uses both cores. :o

I'll be attempting XVID with meGUI in about 11hours, will see if it's any different and what sort of encode speeds I get out of it. Multicore encoding is VERY nice I have to say. I actually have the video compressed from that movie with x264 but need to figure out the soundtrack still. Not had good luck with xport's commandline yet. Keep hoping Madshi will get eac3to working as well for BD as he did for HD-DVd.:bowdown: In any case with just one core it will be interesting to see if XVID encode speed compares with x264 and compare the video quality.:rock:

Adbear

16th March 2008, 12:26

I've made a few 720p X264's using megui, and it always uses all the cores. I've been using it on a Dual Xeon 3GHZ Core 2 Quad system, first pass only uses around 25% of each core, but comes in around 25-26 fps, second pass takes around 3 hours and uses all 8 cores at 100%. It's only when I'm trying to make WMV's that it only uses 4 cores on 1 CPU

BLKMGK

16th March 2008, 20:38

Now THAT is a system to envy!:bowdown: I'd imagine that the low usage on the first pass is I/O bound processing. A problem I'd love to have! Cannot quite swing a dual CPU board I'm afraid. I'm patient though, a quad might be enough for now. If I already didn't need to build a box I'd just live with it but having one just dump it's time. I'll probably move the water cooling system from the old one to my current box too and get it back to stable @4Ghz.

FWIW I've seen code to allow multiple machines to do X264 encodes. I didn't look closely into it as it looked like a bit more complexity than I was looking for at the time but others might be interested, I'll try to look up the name of it.

Tegeril

17th March 2008, 03:17

I would experiment on a few minute clip instead of working with a whole movie.
What speed do you get encoding to H264? 2-3fps?

Quad core processors will definitely increase his 1.9fps. I use a quad (QX6700 stock) to do 12mbit x264 encodes from HDDVD EVOs right now and on pass one, I typically get ~20fps, pass 2 gets ~4-5fps, varies by movie. Though I'm running all kinds of custom flags into x264.

GeeForce11

17th March 2008, 10:47

BLKMGK

17th March 2008, 16:49

Quad core processors will definitely increase his 1.9fps. I use a quad (QX6700 stock) to do 12mbit x264 encodes from HDDVD EVOs right now and on pass one, I typically get ~20fps, pass 2 gets ~4-5fps, varies by movie. Though I'm running all kinds of custom flags into x264.

I think part of the reason why the framerate was SO low was a hung process eating up one core thanks to meGUI hanging while trying to process an audio file. On HD-DVD I generally see 14+FPS on first pass and 3-3.5 on second pass. I too am passing lots of crap to x264 and here's what it looks like:

--pass 2 --bitrate 11000 --stats ".stats" --keyint 14 --min-keyint 2 --ref 5 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --trellis 1 --analyse all --8x8dct --ipratio 1.1 --pbratio 1.1 --vbv-bufsize 14475 --vbv-maxrate 25000 --ratetol 2.0 --qcomp 0.7 --me umh --threads auto --thread-input --progress --no-psnr --no-ssim --output "output" "input" --mvrange 511 --aud --nal-hrd

XVID processed at 8.4FPS and used both cores. I've just now used xport, which I hate BTW, to pull audio from the original MTS and will run it through eac3to and then MKVmerge so I can compare quality. I've only got a 47inch screen so I might not see a big difference. Very much looking forward to seeing this movie in HD finally!:clap:

Tegeril

18th March 2008, 02:12

I ran a little comparison test on a 15min m2ts file, and the X264 encoded MP4 with meGUI seems like lower quality than the XviD encoded AVI with VDubMod at the same 5Mbps bitrate. Any similar experiences?

You can be relatively certain that if you're seeing worse video quality from x264 that something is going wrong during the process or x264 could be passed specific flags that would help. There is no reason that DivX should look better.

BLKMGK

18th March 2008, 22:36

Well, about 85% of the way through my x264 encoded movie it went to hell:bang: Hung XBMC, constant "buffering" messages, it pretty much died. I went back and tried the XVID version and really saw no obvious vid differences although it was a larger file - it was a "fast" encode. Fast forwarding trying to get to where I'd been in the other file failed however, likely a player issue as XBMC is still under development and that was a looong way to forward. I'll try this again tomorrow, I really enjoy this movie:p One thing I noticed was that both cores were maxxed out with the x264 encoded file which is odd. Most of my HD-DVD rips don't go past 85% on the cores except for one scene in Planet Earth that is nutz - but most of this movie really pushed my system and I used the same encoding profile. I even had some dropped frames which is pretty unheard of on my system, very odd!

I'm doing an HD-DVD rip with XVID now, we'll see how it looks. Pretty sure I get dual core decoding with that too so I'll be interested to see the CPU usage. I'll try the BD rip again tomorrow too to see if I can get past that spot.:(

GeeForce11

19th March 2008, 09:33

BLKMGK

19th March 2008, 19:10

What bitrate did you use for the 2 encodings, and which programs did you use? When I play back the encoded video, my dual core never goes above 25%. Are you sure that the system is stable and doesn't make errors during encoding?

11K is the bitrate target as I recall, commandline is above. My CPU usage is usually not this high, it's pretty strange. System is stable for sure, done MANY HD-DVD rips with no issues. Is funny, I'm watching an XVID encode from HD-VD of Batman Begins that came in at just 2Gigs and it looks DAMNED good:rock: I seldom rip files down this small, might need to reevaluate. Bugs me that BD is so damned screwy by comparison:bang: I can and do rip HD-DVD with ease now, this BluRay has been nothing but trouble. Now that SlySoft's tool can rip BD+ I want to switch over but so far this has been a PITA. I just need to keep trying different combinations I guess.

Oh, I used TSremux to make a single M2TS as I recall. Xport to split off the vid and audio. VdubMod's AVSynth script I used with meGUI to encode the files with x264 and also tried XVID - not watched the XVID through yet. The audio was run through eac3to and then the two files muxxed together with MKVmerge. That was about it:clap:

GeeForce11

20th March 2008, 00:25

BLKMGK

20th March 2008, 22:39

I usually encode XviD with 6.4Mbps, and get 4-5GB files at the end for each movie.

Try to join the m2ts files with TSmuxer instead of TSremux, I've read people complaining about glitches in the file when joined with TSremux. Are you sure that the BD wasn't BD+?

Actually, I think you advised me to use TSmuxer and I used that the second time around - I may try it again just to be sure. I am positive it's not BD+, I've got clear video - when it's working.:bang: Honestly I do not mind larger files if I keep the fidelity of the file - that's my primary goal. I don't mind a 9-12Gig file but 40Gigs is nutz!

Won't be able to work on it more until Sunday night, it will be compressing while I'm away. Having issues FFWD what I've got right now - hits a point and locks up. Could be the player, could be the video. On the plus side I've built a 3.1+Ghz quad. I need to load it up but it's going to be crunching video here soon:rock:

Rusty257

28th March 2008, 13:36

BLKMGK

28th March 2008, 16:38

So which method of compressing your EVOs are yall using. I have my own method that i have posted a couple times but the way you all make it sound it is much easier then that. I do not use AVIsynth at all mainly becuase anything with a command line i try to avoid. It reminds me of the old programming days on the apple....10 goto 20...blah blah.

AVIsynth doesn't require a commandline - all I do is open notepad and plug in the correct information for the movie I'm compressing and that's all I do where AVIsynth is concerned.

eac3to DOES use a commandline but it's pretty easy to use and does the majority of the heavy lifting. eac3to pulls out the audio and converts it to the format I choose and it pulls the video out into a MKV container. I then use meGUI and the AVIsynth file I've created to compress the MKV file. Lastly I use MKVmerge to mux the audio and video together - done. I've written it up here (http://forum.doom9.org/showthread.php?t=135361) and tried to give credit where it's due on the Doom9 forums.

BluRay is the big issue for me right now. M2TS is the nightmare for me right now. The files are heyuge, the M2TS format isn't so well supported, and the BD videos that do the seamless scene changes are a PITA. eac3to is slowly supporting this format too but Madshi has his hands full with all of it's quirks - he's making progress though. Sadly HD-DVD arten't going to be around too much longer. I'm buying up the ones I can when at the right price but no more new movies it looks like - booo!:bang: