lumkichi

 I've seen a Windows solution like what I've produced here, created by someone else many years ago - but I don't know what it was called, nor where to get it anymore. I also think it was only for AVI files. I've resorted to writing my own from scratch to handle MP4's. I'd like to share this with anyone and everyone who are interested as this site has helped me a lot to get me to this point.

I've written a DOS batch script in my Windows 10 that can take multiple *mp4 files (and yes - with spaces in the name) and normalize the volume using mp3gain. (I tried MP4GAIN, but the trial period ended in like 1 hour). This one will process much quicker than MP4GAIN, I found out. I do this because I can specifically control ffmpeg and mp3gain every step of the way and do things that mp4gain or ffmpeg-normalize might not be able to.

By all means, it's not fool-proof, nor is it optimized. Rather it can be a springboard for someone to tune/tweak for their own purposes. I extract the audio as mp3 VBR and let mp3gain do its magic on it and then put it back. I find I can process 5 concurrent normalizations, but it's subject to the available hard disk space if you want to try more at once.

Create the folders - D:\Vids\bin can be anywhere, but have the original, working and completed folders inside.

D:\Vids\bin

D:\Vids\bin\original

D:\Vids\bin\working

D:\Vids\bin\completed

Download the latest - they're free as of 9/1/2020 that I've checked. I'm using the following:

D:\Vids\bin\ffmpeg.exe  ( ffmpeg version git-2020-08-28-ccc7120 )

D:\Vids\bin\mp3gain.exe ( mp3gain version 1.4.6 )

Create a D:\Vids\bin\normalize.bat and put this in there:

echo off

setlocal enableextensions enabledelayedexpansion

for %%I in (original\*.mp4) DO (

  echo Creating working\%%~nI.bat

  echo echo off > "working\%%~nI.bat"

  echo echo Unpacking audio and video from %%I . >> "working\%%~nI.bat"

  echo echo ..\ffmpeg.exe -v 0 -i "..\%%I" -q:a 0 -map a ".\%%~nI_audio_only.mp3" -vcodec copy -map 0:v ".\%%~nI_video_only.mp4" >> "working\%%~nI.bat"

  echo ..\ffmpeg.exe -v 0 -i "..\%%I" -q:a 0 -map a ".\%%~nI_audio_only.mp3" -vcodec copy -map 0:v ".\%%~nI_video_only.mp4" >> "working\%%~nI.bat"

  echo echo . >> "working\%%~nI.bat"

  echo echo Normalizing extracted audio with Mp3Gain. >> "working\%%~nI.bat"

  echo echo ..\mp3gain.exe ".\%%~nI_audio_only.mp3" >> "working\%%~nI.bat"

  echo ..\mp3gain.exe ".\%%~nI_audio_only.mp3" >> "working\%%~nI.bat"

  echo echo Reconstructing normalized video from component parts. >> "working\%%~nI.bat"

  echo echo .\ffmpeg.exe -v 0 -i ".\%%~nI_video_only.mp4" -i ".\%%~nI_audio_only.mp3" -codec copy "..\completed\%%~nI%%~xI" >> "working\%%~nI.bat"

  echo ..\ffmpeg.exe -v 0 -i ".\%%~nI_video_only.mp4" -i ".\%%~nI_audio_only.mp3" -codec copy "..\completed\%%~nI%%~xI" >> "working\%%~nI.bat"

  echo echo Post-process clean up. >> "working\%%~nI.bat"

  echo del ".\%%~nI_audio_only.mp3" >> "working\%%~nI.bat"

  echo del ".\%%~nI_video_only.mp4" >> "working\%%~nI.bat"

  echo echo Done. >> "working\%%~nI.bat"

  echo echo on >> "working\%%~nI.bat"

  

)

pause

echo on

Place the original mp4's into your D:\Vids\bin\original folder and run normalize.bat. I do it from the cmd.exe DOS prompt, but you can double-click the batch file in Windows Explorer too. It will search the original folder for mp4's and create individual batch files in D:\Vids\bin\working. You can kick off one, two or four of these batch files in D:\Vids\bin\working at once.

Each original mp4 will be unpacked into a mp4 with no audio, and an mp3 VBR with no video.

The video and audio extractions are done at the same time, thinking this will likely produce two files that can be merged back together later without the video or audio being shorter than the other. The video and audio extracts go into D:\Vids\bin\working.

Then mp3gain.exe will be run against the extracted mp3 and then the audio and video streams will be merged again with ffmpeg and placed into the D:\Vids\bin\completed folder.

The temporary unpacked mp4 and mp3's will be deleted once the process completes.

=======

LIMITATIONS:

The filename of the mp4's cannot contain certain special characters (like ! or ♫) but characters such as spaces ( ) [ ] # ' @ - _ are OK. If you find that the generated script fails or exits much too quickly and doesn't process the video file - try renaming the original file.

I don't have a check for ffprobe to see if the embedded audio is already an mp3, and I found that not all aac audio can be normalized by aacgain.exe. Hence I force everything to come out as mp3 VBR at the highest quality. If the original audio of sufficient quality, I don't mind doing this.

REQUEST

If people can extend this script further to make it aware of the various audio types and do an audio copy if it is already mp3, it would be helpful. Or recognize that the AAC audio needs to be converted to mp3 or left alone as AAC.

Make it handle other video file types (Avi, mkv, wmv, etc.)

Check if a failure occurred after each step and exit the script, and allow for a resume feature so that re-extractions do not need to occur.

Add flags to enable compressing the video file using libx264 -crf 18 -preset veryslow to save some space - but this will make things run very slow.

Any other enhancements you can think of.

~Lum

I'd like to share what I have worked on. 

I have recently acquired a Synology DS220J and installed a Plex server on it.  After being dismayed because I ran into a lot of "Server does not have the power to convert this video" messages, I figured out that this happens to videos that are:

• Not Video Encoded with H.264 (e.g: VP9, H.265, etc.)
• Resolution in excess of 1920 x 1080p
• Audio that is not mp3 or aac (e.g.: flac, ogg, etc.)

So I have written a windows batch script that will bulk-convert my video files (mkv, avi, webm, mp4, flv, etc.) so that the resultant videos will have the following characteristics:

• Video Encoded with H.264
• Resolutions in EXCESS of 1920 x 1080p scaled down to 1920 x 1080p.  Any resolution less than or equal to 1920 x 1080p will be left as-is.
• Video output as *.mp4 regardless of input video type
• Audio converted to aac regardless of input audio type
• Audio gain normalized using aacgain.exe (alternately, we could force things to be mp3 and use mp3gain.exe)

You will need to create the following folder structure and add the following programs:

    Folders

    D:\Vids\bin

    D:\Vids\bin\original

    D:\Vids\bin\working

    D:\Vids\bin\completed

    

    Files

    D:\Vids\bin\ffmpeg.exe

    D:\Vids\bin\ffprobe.exe

    D:\Vids\bin\aacgain.exe

    D:\Vids\bin\normalize_video_for_plex.bat   (copy the batch script below and name it thusly)

Place your video files in the D:\Vids\bin\original folder and run the following "normalize_video_for_plex.bat" DOS/Windows batch file.  Copy/Paste the following DOS/Windows batch code into a text file named "normalize_video_for_plex.bat" and save it.

    rem echo off

    setlocal enableextensions enabledelayedexpansion

    set A=0

    set /A COUNTER=COUNTER+1

    echo on

    for %%I in (d:\Vids\bin\original\*.*) DO (

        

    

        set ORIG_FILE=%%~nI%%~xI

    set TEMP_INPUT_FILE=%%~xI

    set TEMP_WORK=.bat

   

        

        rem echo Creating working\.bat

    echo echo off

    echo cls

    echo set ORIG_FILE="%%~nI%%~xI"

    echo set TEMP_INPUT_FILE=TEMP_!COUNTER!%%~xI

    echo set NEW_FILE="..\completed\%%~nI.mp4"

    echo.

    echo echo -------------------------------------------------------------------------

    echo echo Normalizing: %%I

    echo echo -------------------------------------------------------------------------

    echo.

    echo if not exist "%%I" ^(  

    echo   echo No such file "%%I"  

    echo   goto end

    echo ^)  

    echo.

    echo echo Making local copy as ..\working\%%TEMP_INPUT_FILE%%" 

    echo copy "%%I"  "D:\Vids\bin\working\%%TEMP_INPUT_FILE%%" 

    echo echo -------------------------------------------------------------------------

    echo echo.

    echo. 

    echo for %%%%f in ^("%%TEMP_INPUT_FILE%%"^) do set OUTPUT_AUDIO_ONLY=%%%%~nf_audio_only.aac

    echo for %%%%f in ^("%%TEMP_INPUT_FILE%%"^) do set OUTPUT_VIDEO_ONLY=%%%%~nf_video_only.mp4

    echo for %%%%f in ^("%%TEMP_INPUT_FILE%%"^) do set OUTPUT_FILE=%%%%~nf_new.mp4

    echo. 

    echo rem Set defaults, clean up folder before starting

    echo set VIDEO_SCALE=default

    echo set VIDEO_ENCODING=default

    echo. 

    echo rem MAX_PIXEL_SIZE will equal 1920 x 1080 ^(2073600 pixels^)

    echo set /A MAX_PIXEL_SIZE=1920*1080

    echo. 

    echo if exist "%%OUTPUT_VIDEO_ONLY%%" del /F "%%OUTPUT_VIDEO_ONLY%%"

    echo if exist "%%OUTPUT_AUDIO_ONLY%%" del /F "%%OUTPUT_AUDIO_ONLY%%"

    echo.

    echo echo Determining encoder type and video scale

    echo FOR /F "tokens=* USEBACKQ" %%%%g IN ^(`D:\Vids\bin\ffprobe.exe -v error -select_streams v:0 -show_entries stream^^=width^^,height -of csv^^=s^^=x:p^^=0 "%%TEMP_INPUT_FILE%%"`^) do ^(SET "VIDEO_SCALE=%%%%g"^)

    echo FOR /F "tokens=* USEBACKQ" %%%%g IN ^(`D:\Vids\bin\ffprobe.exe -v error -select_streams v:0 -show_entries stream^^=codec_name -of default^^=nokey^^=1:noprint_wrappers^^=1  "%%TEMP_INPUT_FILE%%"`^) do ^(SET "VIDEO_ENCODING=%%%%g"^)

    echo.

    echo rem Convert VIDEO_SCALE in the form of '1920x1080' to '1920*1080' ^(substitute * for x^)

    echo rem so that it can be evaluated mathematically. 

    echo set PIXEL_EQN=%%VIDEO_SCALE:x=*%%

    echo.

    echo rem Evaluate the pixel size of the current video's single frame

    echo set /A PIXEL_SIZE="%%PIXEL_EQN%%"

    echo echo Found Video encoder '%%VIDEO_ENCODING%%' @ %%VIDEO_SCALE%% ^(%%PIXEL_SIZE%% pixels^)

    echo.

    echo rem If we determine that the current video's PIXEL_SIZE is greater than the MAX_PIXEL_SIZE

    echo rem for 1920x1080, then set the NEED_SCALING flag to downconvert to 1920x1080

    echo if %%PIXEL_SIZE%% GTR %%MAX_PIXEL_SIZE%% ^(set NEED_SCALING=1^) else ^(set NEED_SCALING=0^)

    echo.

    echo rem If the VIDEO_ENCODING is not equal to h264 ^(libx264^) then set NEED_ENCODING flag

    echo if %%VIDEO_ENCODING%% NEQ h264 ^(set NEED_ENCODING=1^) else ^(set NEED_ENCODING=0^)

    echo.

    echo echo.

    echo if %%NEED_SCALING%% EQU 1 ^(

    echo   if %%NEED_ENCODING%% EQU 1 ^(

    echo echo Need to re-encode to h264 and rescale to 1920x1080

    echo   ^) ELSE ^(

    echo echo Need to rescale to 1920x1080 only

    echo   ^)

    echo ^) ELSE ^(

    echo   if %%NEED_ENCODING%% EQU 1 ^(

    echo echo Need to re-encode to h264 only

    echo  ^) ELSE ^(

    echo echo need no changes to video

    echo  ^)

    echo ^)

    echo echo -------------------------------------------------------------------------

    echo echo.

    echo.

    echo echo Extracting video and audio streams to temp files

    echo if %%NEED_SCALING%% EQU 1 ^(

    echo   if %%NEED_ENCODING%% EQU 1 ^(

    echo echo Extracting audio stream to %%OUTPUT_AUDIO_ONLY%%

    echo D:\Vids\bin\ffmpeg.exe -v 0 -stats -i "%%TEMP_INPUT_FILE%%" -q:a 0 -map a -q:a 2 "%%OUTPUT_AUDIO_ONLY%%" 

    echo echo.

    echo echo Extracting video stream to %%OUTPUT_VIDEO_ONLY%%

    echo D:\Vids\bin\ffmpeg.exe -v 0 -stats -i "%%TEMP_INPUT_FILE%%" -an -c:s copy -c:v libx264 -vf scale=1920:1080 -crf 20 "%%OUTPUT_VIDEO_ONLY%%" 

    echo   ^) ELSE ^(

    echo echo Extracting audio stream to %%OUTPUT_AUDIO_ONLY%%

    echo D:\Vids\bin\ffmpeg.exe -v 0 -stats -i "%%TEMP_INPUT_FILE%%" -q:a 0 -map a -q:a 2 "%%OUTPUT_AUDIO_ONLY%%" 

    echo echo.

    echo echo Extracting video stream to %%OUTPUT_VIDEO_ONLY%%

    echo D:\Vids\bin\ffmpeg.exe -v 0 -stats -i "%%TEMP_INPUT_FILE%%" -an -c:s copy -vf scale=1920:1080 -crf 20 "%%OUTPUT_VIDEO_ONLY%%" 

    echo   ^)

    echo ^) ELSE ^(

    echo   if %%NEED_ENCODING%% EQU 1 ^(

    echo echo Extracting audio stream to %%OUTPUT_AUDIO_ONLY%%

    echo D:\Vids\bin\ffmpeg.exe -v 0 -stats -i "%%TEMP_INPUT_FILE%%" -q:a 0 -map a -q:a 2 "%%OUTPUT_AUDIO_ONLY%%" 

    echo echo.

    echo echo Extracting video stream to %%OUTPUT_VIDEO_ONLY%%

    echo D:\Vids\bin\ffmpeg.exe -v 0 -stats -i "%%TEMP_INPUT_FILE%%" -an -c:s copy -c:v libx264 "%%OUTPUT_VIDEO_ONLY%%" 

    echo   ^) ELSE ^(

    echo echo Extracting audio stream to %%OUTPUT_AUDIO_ONLY%%

    echo D:\Vids\bin\ffmpeg.exe -v 0 -stats -i "%%TEMP_INPUT_FILE%%" -q:a 0 -map a -q:a 2 "%%OUTPUT_AUDIO_ONLY%%" 

    echo echo.

    echo echo Extracting video stream to %%OUTPUT_VIDEO_ONLY%%

    echo D:\Vids\bin\ffmpeg.exe -v 0 -stats -i "%%TEMP_INPUT_FILE%%" -an -c:s copy -vcodec copy -map 0:v  "%%OUTPUT_VIDEO_ONLY%%" 

    echo   ^)

    echo ^)

    echo echo -------------------------------------------------------------------------

    echo echo.

    echo.

    echo echo Reassembling converted video and audio files

    echo D:\Vids\bin\ffmpeg.exe -v 0 -stats %%OVERWRITE_FILE%% -i "%%OUTPUT_VIDEO_ONLY%%" -i "%%OUTPUT_AUDIO_ONLY%%" -codec copy "%%OUTPUT_FILE%%"

    echo echo -------------------------------------------------------------------------

    echo echo.

    echo.

    echo echo Normalizing audio track volume

    echo D:\Vids\bin\aacgain.exe "%%OUTPUT_FILE%%"

    echo echo -------------------------------------------------------------------------

    echo echo.

    echo.

    echo echo Moving file to completed folder

    echo move "%%OUTPUT_FILE%%" ..\completed\%%NEW_FILE%%

    echo echo -------------------------------------------------------------------------

    echo echo.

    echo.

    echo echo Cleaning up

    echo if exist "%%OUTPUT_VIDEO_ONLY%%" del /F "%%OUTPUT_VIDEO_ONLY%%"

    echo if exist "%%OUTPUT_AUDIO_ONLY%%" del /F "%%OUTPUT_AUDIO_ONLY%%"

    echo if exist "%%TEMP_INPUT_FILE%%" del /F "%%TEMP_INPUT_FILE%%"

    echo.

    echo :end

    echo echo -------------------------------------------------------------------------

    echo echo.

    echo echo Done

    echo echo on

   

   

    set /A COUNTER=COUNTER+1

    ) > "working\TEMP_!COUNTER!.bat"

    

    echo off

This will create a series of temporary batch scripts in the D:\Vids\bin\working folder named TEMP_1.bat, TEMP_2.bat, etc.  One for each of the videos in the original folder, and they will be the ones to actually perform the conversions.

To begin conversion, simply execute the TEMP_1.bat which will:

1. make a local copy of the video in the working folder (named e.g.: TEMP_x.mp4)
2. analyze the video to determine resolution and encoding and determine if any conversion is necessary
3. extract the audio to TEMP_x_audio_only.aac
4. extract the video to TEMP_x_video_only.mp4 with any re-encoding (if necessary) applied
5. reconstruct the video from the above video and audio files
6. apply volume gains using aacgain.exe
7. write the completed file to D:\Vids\bin\completed
8. clean up the D:\Vids\bin\working folder

I have been bulk-converting my videos for some time now, and this works like a champ and plays well with my Synology DS220J.  I hope it helps you as well.  If you need to make changes in max resolution or audio encoding or any Quality or Performance tweaks or even the install locations, go ahead!  Most of the hard-work has been done for you.

Links for free software used:

• ffmpeg.exe and ffprobe.exe : https://ffmpeg.org/download.html
• aacgain.exe                : http://aacgain.altosdesign.com/
• outube-dl.exe             : https://youtube-dl.org/

~Lum

Labels: ffmpeg.exe ffmpeg aacgain mp4 convert downscale DS220J Plex

I don't know if this warrants a new blog, but the other Sony blog is dated for the 16th. This piece of news has occurred on the 18th.

Sony is moving fast (as they should be) in their CD buyback program. They have partnered with Amazon and UPS (apparently) to buy back your CD's. Amazon purchasers have been getting an email from Amanzon with this incredible offer to buy back their CD.

I'm really not impressed at the overall tone of the exchange program page. The title says it all: "Welcome to the Sony BMG XCP Exchange program" it says, in big blue bold letters. "Why thank you," I'd probably say in a cheerful voice.

But they're not apologetic. Nothing on that page says they are. They're probably saying, "shit, I've got to slap on a fake smile while I spend millions to shut you whiners up." Hell, they're a big company; they have the clout to throw their weight around.

This page has an interesting statement, "you will have the option of selecting whether you would like to receive MP3 files of the title(s) in addition to your replacement CD(s)."

Huh? .... *REALLY* Well then Mr. Sony, load 'em up, pleaze. DRM-protected MP3's? I know of no such types. Maybe 64kbit versions (for faster internet downloads, they might say) of lower audio quality? Hmmmmm. We'll have to see how this plays out.

Rootkits, Spyware Tactics, Copyright Infringements and LPGL Violations.

I grew up with Sony, in the land where Sony originates. The CEO of Sony lived in the same condo as I did--I used to watch his majestic limousine show up every morning, pert and obedient, awaiting to take his master to his domain. I was a commoner (a foreigner at that), who walked 17 minutes to the trainstation to begin my 1.5 hour commute to school. Back then, there was no concept of Sony Japan. It was just..., "It's a SONY!"

My precious microphones were Sony, my first Walkman (circa 1979) was a Sony. Oooh, the anodized blue and silver body with two headphone outputs, a mic and the orange Hot Talk button so you could talk to each other while listening to the Bee Gees (yea, yea). My $600.00 turntable was a Sony (magentically levitated linear tracking arm with a PLL controlled brushless DC motor. The whole platen was the motor! The body was well heavy, concrete I think, with vibration dampening feet. I still have it.

Japan has two government channels (NHK) which provides very professional, unbiased news, educational content and drama series, including award winning TV-series Silkroad. NHK primarily uses Sony equipment--everywhere.

Sony was ingrained into my young psyche. It was a symbol of Japanese staunchiness. It ranked up there with Toyota, Mazda and Honda as an international power to be reckoned with. Sony was it--if you didn't have Sony then, you were nothing.

But it seems that Sony has just gone a little too far for its own good, and for the good of the public. Recently Sony has been distributing CD media bundled with a controversial piece of DRM software. DRM (or this link), in of itself, is not a bad thing. It serves to help protect the interests of the recording industry and the music artists to which it caters.

But Sony has taken this a step further, and in my opinion, several steps too far. Sony's DRM code burned onto a number of undisclosed CD's will attempt to install itself to the user's PC, when it's played on that machine.

Using advanced techniques to circumvent security, the installed code obtains higher security permissions than the user has and installs itself into the very core of the Windows user's operating system. It replaces core files with its own and employing stealth techniques known only to the savviest viruses, it hides itself in such a manner that even the system administrator cannot find it, nor remove it even if he did. What's worse, all of this occurs without the knowledge of the user, apart from the horrible EULA that mentions none of this.

This has led the public to label this piece of software as a "rootkit." It is a coveted prize amongst hackers, virus programmers and the denizens of the ether-netherworld. It's a bane of the good citizens of the computing world, however.

Once embedded into the OS it lies there, quietly monitoring every keystroke, event, password, music that's been played. Occasionally, it wakes up to enforce the DRM policies. But more disturbing, it surreptitiously reports back to the "mothership" of the user's activities.

Several firms have pointed out this also poses a security risk. Of course, Sony denies all of this, stating that this program does nothing bad nor does anything to get in the way, and heaven forbid--violate the user's privacy by sending usage data back to a tracking site, nor does it pose a security threat in any way. Assistant Secretary of Homeland Security, Stewart Baker, made a comment directed at software makers but primarily directed at Sony. "It's very important to remember that it's your intellectual property -- it's not your computer." I agree.

Sony remains unapologetic, and even devalued consumers by making a derogatory statement about them. However bowing to pressure, it made available a means to "bypass" this DRM on November 2, then a service pack2a on November 8, to remove the Cloaking Technology employed by its DRM code. It was only going to decloak the software--not remove the DRM entirely. Critics blew their stack! Here's a nice write up on the "patched" XCP code.

Then around November 10, the first Trojan Horse using Sony's rootkit to infiltrate then cloak itself, appeared. A real slap in Sony's face. Bad news for Sony continued as a second variant of a Trojan Horse that exploits Sony's DRM code was discovered later that day.

This software comes without an uninstaller. Any attempt to remove this software damages your operating system, and employs tactics that makes it difficult to do so in the first place. To remove the software at all, you must contact Sony through various emails, website registrations, click throughs, agreement with other policies, etc. If you're lucky, it might have been removed.

Public outcry has reached cacophonic proportions. Microsoft has reponded with a patch to disable Sony's rootkit. Anti-spyware software companies have capitalized on this and labeled Sony's XCP code as spyware and offer to detect (and disable) it. Links (1, 2, 3, ...)

Crippled Sony has posted this statement on November 16, and is pulling the controversial code from the shelves world wide. It has disclosed a list of 52 song titles which contain the XCP technology, and has provided a FAQ about this rootkit. Interesting though, they still maintain that their XCP only contacts a website for ad-banner rotation despite proof that this thing dials out to connected.sonymusic.com (See a write up on how Dan Kaminsky estimated nearly 500,000 networks were "infected" with XCP Rootkit using this URL).

---

Whew! That's a handful, isn't it? To top it off, it has come to the public's attention that Sony's Rootkit code (XCP) incorporates LGPL protected code, which necessitates Sony to publish the code and the derivative code (the XCP code itself). Sony has published a number of open-source code for a its more obscure products. But none can be found for this rootkit.

Isn't that just icing on the cake? Companies like SonyBMG are no different than the smaller companies like SimpleDevices and OmnifiMedia when it comes to GPL violations and Software Copyright Infringements. I recently mentioned this on an earlier blog (GPL Violations Irk Me) as well...

Chances are, we might not ever see the source code for SimpleDevices nor Sony's controversial DRM. These companies are utilizing sleazy tactics against us, hiding behind beaurcratic mire.

I used to like Sony. In a way I still do. I'm trying to justify that I like Sony, and by saying I dislike Sony BMG which is a different company. But in the end, Sony Americas, Sony UK, Sony BMG, Sony Japan are all..., well Sony. The damage is done.

~Lum

"Compile In Place" is a concept that I've been working for the better part of a year and a half. I do believe other Java programmers are aware of it and many of them probably utilize it. I have searched and searched but never found a name or a reference to this kind of programming.

Simply put, Compile In Place allows a person to compile a single Java file without having the entire source code library. This is particularly powerful when trying to maintain Java programs ehose authors have long since left the company and did not disclose the location of the source.

Normal Programmers

Normally, a programmer would have access to Java Project and begins to compile and build starting with the main file. As the main file is compiled, source files for the referenced objects are checked by the Java compiler and if they are newer (by timestamp) than the associated class files, or if the class file is not present at all, then those referenced classes are compiled as well. Those referenced classes may reference other classes and objects until the entire project is built top down.

Alternately a Java programmer might use an AntScript or an IDE (such as Sun One, JPad Pro) to build his files, compiling each source file independently, from the bottom up.

In any case, the Java compiler likes to have the source file present when compiling, as do compilers of other languages. But maintenance programmers who must reverse engineer their predecessor's code may not have this luxury.

But Me?

A Java application is a collection of class files (and supporting properties files, etc.). Compile in place allows a single Java file to be compiled without having any other source file present AS LONG AS the compiled binary file is present in the proper package structure.

If I have decompiled a single file in myCompany.jar, made modifications to it, I can still recompile my single java file PROVIDED I either reference the original jar file in my CLASSPATH along with all of the library files referenced by myCompany.jar itself, and I maintain the package structure.

Alternately, I can extract the contents of the myCompany.jar file completely, and I place my .java I wish to compile in the appropriate directory (according to the package information), then I can simply invoke the command

C:\java> javac com\myPackage\myUtil\MyMain.java

C:\java> _

And it will compile only the one file: MyMain.java to produce one new file MyMain.class. In this case, the entire application is NOT recompiled even though Main.class might be the main class of the entire project.

Why is This So?

Well, for one, there are no source codes to check for -- what's there is what's there. Additionally Java and Javac use reflection extensively which inspects contents of the class files to check for valid public method signatures and public variables. It does not need the source if the binary class files referenced by my .java program checks out.

Lucky, because my job relies heavily on Compile In Place to effect changes to a piece of software who was written by a disgruntled ex-employee. Without it, I just would not be able to repair the bugs in our company's software, sometimes.

I guess this is like C or Assembly where I can compile a simple .c file or .asm file and let the linker link everything provided the libraries and objects are all present. The difference is that in .c and assembly, the output of a linking process is usually a single executable, which cannot be used to help in compiling another single .c or .asm file. (You know, if I have a bunch of object files, it implies I had a bunch of source files to begin with).

But a pitfall...?

There is a pitfall! It might happen if you've been trying to repair a bug in this code elsewhere and you leave your source code in and you recompile another java file in your project. That new java file might have dependencies on your first piece of code and you might encounter hundreds of compiler errors (actually the compiler would give up after a hundred, usually). It may not go away until you move your first source code out to where the compiler cannot find it.

You'll just have to look very carefully at the error output and determine that it might not be coming from the class you're currently compiling, but instead from another class you were tinkering with.

For more info check out Omnifi Wiki Pages.


~lumkichi

What does it take for Companies to comply with the GPL?

Frustrating it is for the software developer who writes code that is gratiutiously usurped by a software company without any credit or royalty. Frustrating still, for the owners of a product by these companies that want to repair problems and ehance the product but are blocked from doing so by proprietary encryption that prevents access to the GPL code within. Fun, it is, to learn that the companies have all but abandoned any further development of the product too.

By this time many people are aware of the Gnu General Public License (GPL). It is a license created by the Free Software Foundation that grant any user the right to copy, modify and redistribute programs and source code from developers that have chosen to license their work under the GPL. The stipulation is that the user release and make available any and all GPL-licensed source code. Further all derivitives of the GPL-source code also becomes covered under the GPL. That is, any user that modifies a piece of GPL code to work with his project (or product), that entire project may be subject to the GPL and the source code must be released to the public.

So what happens when a company builds and sells a product that incorporates GPL code? Well, they're allowed to use and sell that product for a profit. BUT, they're also required to release all of the source code utilized as well as any and all modified GPL source code. If a company takes GPL source and modifies it heavily with their "proprietary" code, then that new piece of code is considered a derivative of the original GPL code and thus automatically covered by and subject to the terms of the GPL, and must be made public.

What companies fail to realize is that is by statically binding GPL code into their product, they have now created an entire project that is subject to the GPL, and the entire source code must be released, in its re-compilable form. What's surprising is that large, well-known companies are brought into the spot-light for these transgressions. To Wit:

Linksys (WRT54G): http://developers.slashdot.org/article.pl?sid=03/07/31/1350217&tid=117&tid=193&tid=8&tid=106
Nvidia : http://slashdot.org/features/00/05/01/0047219.shtml
Compaq : http://slashdot.org/article.pl?sid=00/09/13/1540253&tid=140
Epson : http://slashdot.org/article.pl?sid=02/09/11/2225212
Various companies : http://www.drunkenblog.com/drunkenblog-archives/000534.html

Linksys eventually relented and published a complete set of source code for the WRT54G.

The issue is that this is not new, nor is it relegated to large companies. Other companies such as Rockford Fosgate/Omnifimedia who distribute two versions of media players (one in-home streaming media, one in-car media player) from SimpleDevices are in the same boat.

Members of the Omnifi Yahoo forum have learned that companies often play naive or take the "innocent" tact. The DMP1 & DMS1 players manufactured and distributed by SimpleDevices, OmnifiMedia and Rockford Fosgate are built on a Linux kernel. A couple of tech-savvy forum members were able to peer into the the kernel and applications and discovered massive amounts of GPL code. They approached Rockford Fosgate, SimpleDevices Omnifimedia about a possible GPL violation. The companies just shrugged their shoulders and said it was the other company's problem. Finally someone at SimpleDevices.com came clean and published a list of source code (in their pristine unmodified condition) to some (not all) of the open source modules used in the system. Additiionally, the manufacturers seem to have statically incorporated GPL code into their binaries. Members of the forum have set up and signed an online petition against the software companies: http://www.petitiononline.com/omnifi/ . So far, they're not getting anywhere.

Other possible violations in action:

Checkpoint: http://www.ethereal.com/lists/ethereal-dev/200308/msg00175.html
Cygwin: http://www.cygwin.com/ml/cygwin/2003-03/msg01926.html
DrDos: http://freedos.org/freedos/news/bits/drdos81.html

What does it take to get these companies off their duffs and onto the chopping block?