I’ve got 113 reels of microfilm I’d like to digitize and I’m looking for a ballpark estimate for the project.

Here’s the info I know, please let me know if you need anything else:

• There are an average of 600 images per reel (about 67,800 images)
• I’d like to scan at 300 dpi, 8bit grayscale lossless images (tiff? png?)
• I have the copyright on the images on these reels
• The reels are lightly used duplicates of the master reels. The master reels are unfortunately unavailable
• The images are all scanned newspapers
• I don’t need any OCR done
• The only metadata I need is which reel each of the images came from eg. One directory per reel with incremental file names would be just fine.

Project Background: I would like to put 128 years of Iron County Miner newspaper archives online. They would be freely available (no subscription or account required) and there’s no plan to make money from them. Since there’s no revenue expected I’m looking for ways to reduce costs while still putting something out there to benefit genealogists and historians.

The master rolls are held by the Wisconsin Historical Society who wants nearly $10,000 ($0.145/image) for the project or $80 per reel to send us fresh copies of the reels. From their perspective, I think that’s probably fair; they aren’t in the digitization business and they probably aren’t set up to do this sort of project in a streamlined manner. They also can’t amortize their digitization equipment costs across so many clients as a commercial company can.

For me though, $10,000 means that I can’t pursue this project right now.

Most digitization companies I have contacted have been reluctant to provide even a ballpark quote without seeing test reels, and I understand that that is a factor. Right now though, I just need a gauge to determine if this project is viable. If $10,000 is the real cost for this sort of project it will have to wait till I’m rich, but if I can get a cheaper quote I hope to make it happen this summer.

Pre-Announcing NewspaperCMS

I have been working on a CMS (Content Management System) called NewspaperCMS, to host the scanned images with and to make them easily navigable. It is licensed under the GPLv2 so anybody needing to host newspaper archives can use it.

Here’s its page on Google Code: http://code.google.com/p/newspapercms/

I would classify it as in late Alpha or early Beta stages right now. I’ll do an official post on it as it matures and as I get a publicly accessible test site set up. As a teaser, features include:

• Browse collection by microfilm, newspaper or date
  • Drill down within those categories by newspaper, issue, year or month
• Access-driven generation of midsized images. No need to generate 60,000 midsized images ahead of time.
• Valid HTML5/CSS3
• HTML5/Canvas based client-side image viewer. The user can zoom, rotate, invert, sharpen and change the contrast of the image (uses the http://www.pixastic.com/JavaScript libraries)
  • Falls back to a static image if they don’t have Canvas or JavaScript support
• Built in search engine
• Support for the tesseract OCR engine

As I said, it’s still in development, but if you need something like it, you can play with it now. It’s at the point where more development doesn’t make sense until I know I can get the microfilms scanned.

Zooming in on the image quickly causes the canvas to outgrow my browser window. The div around the canvas is set to use overflow: auto so that the growing canvas doesn’t disrupt the rest of the page flow.

<div id="pageimgdiv" style="max-width: 100%; overflow: auto;">
   <canvas>
      Your browser doesn't support HTML5 Canvas
   </canvas>
</div>

The overflowed div gains horizontal scrollbars (but not vertical ones, since there’s no max-height in my case). Unfortunately many people, including myself, don’t have horizontal scrolling configured for their mouse which means scrolling down to the scrollbar, moving over, then scrolling back up.

JQuery To The Rescue

I was able to use JQuery and the scrollLeft() function to make the canvas dragable within the div. The canvas itself doesn’t change sizes or pan (which would require a second canvas used as a buffer, I think). Instead we get the mouse position and current scrollLeft setting when the mouse is clicked, and then scroll more as they move the mouse, until they release the mouse or until they leave the wrapper div.

I’m using JQuery 1.7.2.

$(document).ready(function(){
    $('#pageimgdiv').on(
    {
	mousedown: function(clicke){
	    origX = clicke.pageX + $('#pageimgdiv').scrollLeft();
	    $('#pageimgdiv').on(
	    {
		mousemove : function(e){
		    curX = e.pageX + $('#pageimgdiv').scrollLeft();
		    var diff = (origX - curX);
		    var newpos = $('#pageimgdiv').scrollLeft() + diff;
		    if(newpos > ($('canvas').width() - $('#pageimgdiv').width())){
			newpos = ($('canvas').width() - $('#pageimgdiv').width());
		    }
		    if(newpos < 0){
			newpos = 0;
		    }
		    $('#pageimgdiv').scrollLeft(newpos);
		}
	    }
	    );
	},
	mouseleave: function(){
	    $('#pageimgdiv').off('mousemove');
	},
	mouseup: function(){
	    $('#pageimgdiv').off('mousemove');
	},
	click: function(){
	    $('#pageimgdiv').off('mousemove');
	}
    }
    );
});

Embrace and Extend

This code only scrolls horizontally. You could easily extend it to use scrollTop() and enable vertical scrolling as well.

Most Xdebug/NetBeans tutorial assume that you’re doing development on your local machine. That’s a fine setup, but not what I was needed for this project.

Environment

Server: A typical Linux server — Debian, Apache2 and PHP.

Debugger: Xdebug

Client: Firefox on Windows, etc.

IDE: NetBeans 7.1.1

Other Tools: PuTTY

Setting Up Xdebug

Install xdebug with the command:

pecl install xdebug

The final lines of output should say something like:

Build process completed successfully
Installing '/usr/lib/php5/20090626/xdebug.so'
install ok: channel://pecl.php.net/xdebug-2.2.

Take note of the install path, /usr/lib/php5/20090626/xdebug.so, in this case. Now add the following to your PHP configuration. I created a new .ini file at

/etc/php5/conf.d/xdebug.ini

Its contents should be as follows (change the zend_extension to match the install path found above):

[xdebug]
zend_extension = "/usr/lib/php5/20090626/xdebug.so"

xdebug.remote_enable = on

; Most users won't want autostart. More on this later.
; xdebug.remote_autostart = on
xdebug.remote_autostart = off
xdebug.remote_handler = dbgp
xdebug.remote_port = 9000
xdebug.remote_server = localhost
xdebug.remote_mode = req

; Most users won't want a hard coded idekey. More on this later.
; xdebug.idekey = netbeans-xdebug
output_buffering = off

xdebug.remote_log = "/var/log/xdebug.log"

Restart Apache to complete the installation.

xdebug.remote_server = localhost. Localhost? Wait a second, I thought this was for working with a remote server? Yes, but xdebug needs to be able to connect to your computer. The easy option is to have xdebug connect to the server’s localhost, then use PuTTY to create an SSH tunnel so that NetBeans can listen on your computer’s localhost.

The harder option is to configure your home or office router to forward port 9000 to you, be sure to never change your IP address, and open port 9000 on your Windows firewall. You could use xdebug.remote_connect_back so that xdebug would connect to whichever IP made the web request, but then someone who isn’t you could access your code.

In my opinion, the SSH tunneling is the cleanest option. You can use it anywhere that you have SSH access, xdebug access is restricted to those who have SSH access and you don’t have to worry about your IP address changing.

Creating an SSH Tunnel for Xdebug

Using PuTTY.exe create and save a new SSH session which connects to your server. In this saved session configure a tunnel. Set the source port to 9000, the destination to localhost:9000, and choose the Remove and Auto radio buttons. Click the Add button to add that port forwarding configuration, then save that session to use every time you want to use xdebug.

Go ahead and connect to that saved PuTTY session now. Once you’re connected to your server you should be able to run netstat to verify that it’s working correctly. It should show something like this:

netstat -a -n | grep 9000
tcp        0      0 127.0.0.1:9000          0.0.0.0:*               LISTEN
tcp6       0      0 ::1:9000                :::*                    LISTEN

Setting up NetBeans

Set up your NetBeans project like you usually would. Verify that the PHP Debugging Settings (In the Tools->Options menu) are set to debugger port 9000, and the Session ID of netbeans-xdebug. I had problems with the Watches and Balloon Evaluation options. YMMV.

In your project properties (Right click your project, click Properties) edit the Run Configuration’s Advanced properties. Select “Do Not Open Web Browser”.

A Note on Path Mappings

Based on the tutorials I read, most users don’t seem to need Path Mappings. NetBeans seems to figure out the path based on the upload directory, the URL and the Web Root settings.  That didn’t work for me. I needed Path Mapping because I had a symlink on the server.

PHP (and so Xdebug) dereferences symlinks and NetBeans needs the mapping between the dereferenced path on the server and the local sources directory.

This is a known issue, but not very well known. Symptoms that you need to  use mapping include NetBeans not breaking in symlinked files, NetBeans not opening the file or not opening the correct file when Xdebug connection is made. Or it may work only if you use “Debug File” instead of “Debug Project”.

 Debugging With Xdebug and NetBeans

Here’s where you have some options. The manual way to active Xdebug is to append XDEBUG_SESSION_START=netbeans-xdebug to your query string when you request a page in the browser. With the current setup, this should work. Go ahead and test it.

Choose Debug -> Debug Project from the NetBeans menu. Then, in your browser, go to http://example.com/path/to/page.php??XDEBUG_SESSION_START=netbeans-xdebug (replacing the URL with your own, of course).

If that doesn’t work then something isn’t set up correctly. It might be NetBeans, it might be Xdebug, it might be the SSH tunnel. Figure it out, get it fixed, then keep reading (you can leave comments here, and I’ll help as I can. Google is pretty helpful too).

If that does work, then GREAT!

Now you have some options.

Browser Extensions

Firefox and Chrome both have Xdebug extensions that set the XDEBUG_SESSION_START parameter in the http headers. This makes it so you don’t have to type it yourself.  Yay Firefox. Yay Chrome.

Always Auto-start Xdebug

If you are testing embedded webkit, or from mobile devices, or something else that’s not a normal browser, then appending XDEBUG_SESSION_START is going to be difficult. This is where I send you back to your xdebug.ini file to change some settings. If you need to debug requests from these sorts of devices, then you’re going to want to edit xdebug.ini and set:

xdebug.remote_autostart = on
xdebug.idekey = netbeans-xdebug

This will cause Xdebug to attempt to connect on port 9000 on every request. The idekey setting is so that NetBeans will know that the connection is for it.

Other PHP Debugging Tools

If you are looking for good PHP debugging tools, you may also want to try out KCacheGrind which will profile your code giving you an idea what’s taking up time and memory. HipHop-PHP, a tool from Facebook of all places, compiles PHP into C++. In the process it spits out all sorts of helpful errors and notices that will help you find errors in your code.

That concludes one more NetBeans Xdebug tutorial that will hopefully get you that much closer to doing some serious PHP debugging. Happy coding!

You might want to always append a trailing slash if you depend on it in $_SERVER['REDIRECT_URL'] or if you want to ensure that content is only available via a single URL for search engines etc.

Here’s the Apache mod_rewrite code:

<IfModule mod_rewrite.c>
RewriteEngine On

# Always append a trailing slash
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !/$
RewriteRule . %{REQUEST_URI}/ [R=301,L]

</IfModule>

The plugin is named Todayish because if there isn’t a blog post from today’s
date in previous years it will use the blog post which is the least
number of days away from todays date.

Todayish in History provides a function for use in themes as well as a
widget for use in your sidebar.

Todayish In History’s Homepage

Todayish In History has a permanent home at http://stuporglue.org/todayish-in-history/.

Hopefully this proves to be a useful plugin for someone. Please send me any feedback. You can use the comments, the contact page, or email me at stuporglue@gmail.com.

Screenshots

Demo

You can see Todayish In History in action at the top of every page over at The Fridley Farmer.

At first I suggested that people needed more robust email parsing use a dedicated library. But no one seemed to want to do the coding for that, so I ended up writing a new version which uses the PEAR mimeDecode.php library to do the parsing.

Without further ado, here’s mailReader.php!

E-mail Processing Script Features:

1. Saves the e-mail sender, subject and body to a database
2. Saves any attachments as files and creates an entry for those files in the database, associated with the e-mail info in #1
3. Sends  a response back to the sender telling them what files were received and their file sizes
4. Checks a list of allowed senders to make sure we only take files from specified addresses.

Database Setup:

If you’re going to use the database features, you’ll need a database. Here’s the SQL to create an identical setup to the one I have:

-- Here's my DB structureCREATE TABLE IF NOT EXISTS `emails` (
  `id` int(255) NOT NULL AUTO_INCREMENT,
  `from` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `subject` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `body` text COLLATE utf8_unicode_ci NOT NULL,
  `date` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=1 ;

CREATE TABLE IF NOT EXISTS `files` (
  `id` int(255) NOT NULL AUTO_INCREMENT,
  `email_id` int(255) NOT NULL,
  `filename` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `size` varchar(20) COLLATE utf8_unicode_ci NOT NULL,
  `mime` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=1 ;

Security

Make sure that your upload directory is out of your webroot. If someone emails you a malicious PHP script (eg. Virus.php) and can access it via the web, they could infect your server or your visitors. Many servers are configured to automatically treat .pl and .cgi as CGI scripts and run them as well. You do not want to create a way for untrusted users to upload files to your webroot!

With the file names in the database you can use Readfile to pass files down to users.

The Script: mailReader.php

Download it here.

#!/usr/bin/php -q
<?php
//  Use -q so that php doesn't print out the HTTP headers

/*
 * mailReader.php
 *
 * Recieve mail and attachments with PHP
 *
 * Usage:
 * This script expects to recieve raw emails via STDIN.
 *
 * Configure your mail server to pipe emails to this script. (See
 * http://stuporglue.org/add-an-email-address-that-forwards-to-a-script/
 * for instructions).  Make this script executable, and edit the
 * configuration options to suit your needs. Change permissions
 * of the directories so that the user executing the script (probably the
 * mail user) will have write permission to the file upload directory.
 *
 * By default the script is configured to save pdf, zip, jpg, png and gif files.
 * Edit the switch statements around line 200 to change this.
 *
 * Requirements:
 * You will need mimeDecode.php from http://pear.php.net/package/Mail_mimeDecode/
 * I used version 1.5.5
 *
 * Copyright 2012, Michael Moore
 * Licensed under the same terms as PHP itself. You are free to use this script
 * for personal or commercial projects. Use at your own risk. No guarantees or
 * warranties.
 *
 * Contact:
 * <stuporglue@gmail.com>
 * http://stuporglue.org
 *
 * Support:
 * Limited free support available in the comments on the webpage for this script
 * or via email. Contracted support available for specific projects.
 * http://stuporglue.org/mailreader-php-parse-e-mail-and-save-attachments-php-version-2/
 *
 * Thanks:
 * Many thanks to forahobby of www.360-hq.com for testing this script and helping me find
 * the initial bugs.
 * Thanks to Craig Hopson of twitterrooms.co.uk for help tracking down an iOS email handling bug.
 */

global $save_directory,$saved_files,$debug,$body;

/*
 *
 * 	Configuration Options
 *
 */

// What's the max # of seconds to try to process an email?
$max_time_limit = 600; 

// A safe place for files WITH TRAILING SLASH
// Malicious users could upload a php or executable file,
// so keep this out of your web root
$save_directory = "/a/safe/save/directory/";

// Allowed senders is now just the email part of the sender (no name part)
$allowed_senders = Array(
    'myemail@example.com',
    'whatever@example.com',
); 

// Send confirmation e-mail back to sender?
$send_email = FALSE; 

// Save e-mail message and file list to DB?
$save_msg_to_db = FALSE; 

// Configure your MySQL database connection here
$db_host = 'localhost';
$db_un = 'db_un';
$db_pass = 'db_pass';
$db_name = 'db_name';

$debug = FALSE;

/*
 *
 * 	End of Configuration Options
 *
 */

//Anything printed to STDOUT will be sent back to the sender as an error!
//error_reporting(-1);
//ini_set("display_errors", 1);

// Initialize the other global, set PHP options, load email library
$saved_files = Array();
set_time_limit($max_time_limit);
ini_set('max_execution_time',$max_time_limit);
require_once('mimeDecode.php');

// Some functions we'll use
function formatBytes($bytes, $precision = 2) {
    $units = array('B', 'KB', 'MB', 'GB', 'TB');

    $bytes = max($bytes, 0);
    $pow = floor(($bytes ? log($bytes) : 0) / log(1024));
    $pow = min($pow, count($units) - 1);

    $bytes /= pow(1024, $pow);

    return round($bytes, $precision) . ' ' . $units[$pow];
} 

// Find a happy place! Find a happy place!
function saveFile($filename,$contents,$mimeType){
    global $save_directory,$saved_files,$debug;
    $filename = preg_replace('/[^a-zA-Z0-9_-]/','_',$filename);

    $unlocked_and_unique = FALSE;
    while(!$unlocked_and_unique){
	// Find unique
	$name = time() . "_" . $filename;
	while(file_exists($save_directory . $name)) {
	    $name = time() . "_" . $filename;
	}

	// Attempt to lock
	$outfile = fopen($save_directory.$name,'w');
	if(flock($outfile,LOCK_EX)){
	    $unlocked_and_unique = TRUE;
	}else{
	    flock($outfile,LOCK_UN);
	    fclose($outfile);
	}
    }

    fwrite($outfile,$contents);
    fclose($outfile);

    // This is for readability for the return e-mail and in the DB
    $saved_files[$name] = Array(
	'size' => formatBytes(filesize($save_directory.$name)),
	'mime' => $mimeType
    );
}

function decodePart($body_part){
    global $body,$debug;
    if(array_key_exists('name',$body_part->ctype_parameters)){ // everyone else I've tried
	$filename = $body_part->ctype_parameters['name'];
    }else if($body_part->ctype_parameters && array_key_exists('filename',$body_part->ctype_parameters)){ // hotmail
	$filename = $body_part->ctype_parameters['filename'];
    }else{
	$filename = "file";
    }

    if($debug){
	print "Found body part type {$body_part->ctype_primary}/{$body_part->ctype_secondary}\n";
    }

    $mimeType = "{$body_part->ctype_primary}/{$body_part->ctype_secondary}"; 

    switch($body_part->ctype_primary){
    case 'text':
	switch($body_part->ctype_secondary){
	case 'plain':
	    $body = $body_part->body; // If there are multiple text/plain parts, we will only get the last one.
	    break;
	}
	break;
    case 'application':
	switch ($body_part->ctype_secondary){
	case 'pdf': // save these file types
	case 'zip':
	case 'octet-stream':
	    saveFile($filename,$body_part->body,$mimeType);
	    break;
	default:
	    // anything else (exe, rar, etc.) will faill into this hole and die
	    break;
	}
	break;
    case 'image':
	switch($body_part->ctype_secondary){
	case 'jpeg': // Save these image types
	case 'png':
	case 'gif':
	    saveFile($filename,$body_part->body,$mimeType);
	    break;
	default:
	    break;
	}
	break;
    case 'multipart':
	if(is_array($body_part->parts)){
	    foreach($body_part->parts as $ix => $sub_part){
		decodePart($sub_part);
	    }
	}
	break;
    default:
	// anything else isn't handled
	break;
    }
}

//
// Actual email handling starts here!
// 

// Process the e-mail from stdin
$fd = fopen('php://stdin','r');
$raw = '';
while(!feof($fd)){ $raw .= fread($fd,1024); }

// Uncomment this for debugging.
// Then you can do
// cat /my/saved/file.raw | ./mailReader.php
// for testing
//file_put_contents("$save_directory/" . time() . "_email.raw",$raw);

// Now decode it!
// http://pear.php.net/manual/en/package.mail.mail-mimedecode.decode.php
$decoder = new Mail_mimeDecode($raw);
$decoded = $decoder->decode(
    Array(
	'decode_headers' => TRUE,
	'include_bodies' => TRUE,
	'decode_bodies' => TRUE,
    )
);

// Set $from_email and check if it's allowed
$from = $decoded->headers['from'];
$from_email = preg_replace('/.*<(.*)>.*/',"$1",$from);
if(!in_array($from_email,$allowed_senders)){
    die("$from_email not an allowed sender");
}

// Set the $subject
$subject = $decoded->headers['subject'];

// Find the email body, and any attachments
// $body_part->ctype_primary and $body_part->ctype_secondary make up the mime type eg. text/plain or text/html
if(is_array($decoded->parts)){
    foreach($decoded->parts as $idx => $body_part){
	decodePart($body_part);
    }
}

// $from_email, $subject and $body should be set now. $saved_files should have
// the files we captured

// Put the results in the database if needed
if($save_msg_to_db){
    mysql_connect($db_host,$db_un,$db_pass);
    mysql_select_db($db_name);

    $q = "INSERT INTO `emails` (`from`,`subject`,`body`) VALUES ('" .
	mysql_real_escape_string($from_email) . "','" .
	mysql_real_escape_string($subject) . "','" .
	mysql_real_escape_string($body) . "')";

    mysql_query($q) or die(mysql_error());

    if(count($saved_files) > 0){
	$id = mysql_insert_id();
	$q = "INSERT INTO `files` (`email_id`,`filename`,`size`,`mime`) VALUES ";
	$filesar = Array();
	foreach($saved_files as $f => $data){
	    $filesar[] = "('$id','" .
		mysql_real_escape_string($f) . "','" .
		mysql_real_escape_string($data['size']) . "','" .
		mysql_real_escape_string($data['mime']) . "')";
	}
	$q .= implode(', ',$filesar);
	mysql_query($q) or die(mysql_error());
    }
}

// Send response e-mail if needed
if($send_email && $from_email != ""){
    $to = $from_email;
    $newmsg = "Thanks! I just uploaded the following ";
    $newmsg .= "files to your storage:\n\n";
    $newmsg .= "Filename -- Size\n";
    foreach($saved_files as $f => $s){
	$newmsg .= "$f -- $s\n";
    }
    $newmsg .= "\nI hope everything looks right. If not,";
    $newmsg .=  "please send me an e-mail!\n";

    mail($to,$subject,$newmsg);
}

if($debug){
    print "From : $from_email\n";
    print "Subject : $subject\n";
    print "Body : $body\n";
    print "Saved Files : \n";
    print_r($saved_files);
}

Thanks

Many thanks to forahobby for testing this script and helping me squash a bunch of little bugs. Thanks to Craig Hopson for his help finding a problem handling emails from iOS devices.

I was getting this error:

PuTTY X11 proxy: MIT-MAGIC-COOKIE-1 data did not matchWarning: This program is an suid-root program or is being run by the root user.
The full text of the error or warning message cannot be safely formatted
in this environment. You may get a more descriptive message by running the
program as a non-root user or by removing the suid bit on the executable.
xterm Xt error: Can't open display: %s
root@myserver:~#

Some googling lead me to this blog post which didn’t work for me, but put me on the right path.

Here’s the line I ended up using

su - non_root_user_name -c "xauth list"  | xargs -n 3 xauth add

If you need to use this frequently you could add an alias to you /root/.bashrc file

alias xwork='su - non_root_user_name -c "xauth list"  | xargs -n 3 xauth add'

That way you can just run xwork when you need X11 working.

Note: All of these tools are Linux command line tools. Less convenient than a GUI, but much more appropriate for bulk operations and scripting.

Preparing the Videos

Tovid can take any video format and make a DVD from it by calling ffmpeg and other libraries when it needs to. In practice, I find that tovid and ffmpeg fight like sibling when they get along it’s true love, but when they fight things get broken.

Encoding is the slowest part of making a DVD, so we’ll do it separately. This way we can tweak the DVD quickly without re-encoding each time. FFMpeg can do it with the command:

ffmpeg -i SourceFile.whatever -target ntsc-dvd outputfile.mpg

This will produce mpeg2 files which don’t need to be further encoded in order to be put on a DVD.

Splitting Videos

If the resulting outputfile.mpg is too big for a DVD, you will need to split it. FFMpeg can split your file using a starting second (-ss) and a time (-t).  With this syntax you can break your file into as many pieces as you need to fit them on a DVD.

ffmpeg -i SourceFile.whatever -ss 0 -t 7200 -target ntsc-dvd outputfile-part1.mpg
ffmpeg -i SourceFile.whatever -ss 7201 -target ntsc-dvd outputfile-part2.mpg

Creating the Metadata

Tovid will want a couple of pieces of metadata, depending on what menu options you choose. Main titles, submenu titles and a disc title.

Since I was digitizing family videos, I wanted the disc title to be the years spanned. So I created a CSV file with 4 columns. the year would become the disc title.

filename,year,long description,short description
SourceFile.whatever,1990,The family goes sledding at Ice Hill,Sledding

The Tovid Wrapper Script

I know Tovid is already a wrapper for dvdauthor, ffmpeg and whatever other tools it uses behind the scenes, but what’s another layer, right?

I put all of my videos, the CSV file and this script in a folder:

#!/usr/bin/env php
<?php

/*
 * @brief Make as few DVDs as needed to fit all of the videos listed in a CSV file
 *
 * Copyright Michael Moore <stuporglue@gmail.com>
 * This script assumes that
 *      * All of the videos have already been encoded into mpeg2 format for DVD
 *      * All of the videos paths are relatie to INPUTDIR
 *
 * As many videos are fit into one DVD as possible. Videos are added in the order
 * listed in the CSV file. If a video is too big for a DVD the script will tell
 * you, and then exit. DVDs are named either "year", "year1 - year2" or
 * "year disc n", using the year from the CSV.
 *
 * Requires php-cli installed so that we can fork processes.
 *
 * Our metadata.csv file format:
 * pathtofile,year,looooooooooooong title here,short title
 *
 * We assume the file has a header row.
 */

//
//         Settings
//
define('DVDSIZE',8500000000); // Our estimate of the number of bytes available on a DVD+R DL
define('MENUOVERHEAD',45*1024*1024); // Generous leeway for the menu system
define('INPUTDIR',"/home/myuser/videos/input/"); // starting point for all file paths. Use / for abs. paths
define('TMPDIR','/tmp/tovid/'); // Where do you want the tmp files?
define('OUTDIR',"/home/myuser/videos/DVDs/"); // Destination?
define('MAXCHILDREN',5); // How many tovids to run simultaneously? Probably 1-2 less than the # of cores available?
$fh = fopen(INPUTDIR.'/metadata.csv','r');

//
//         No Lifeguard on duty!
//
@mkdir(TMPDIR);
chdir(TMPDIR); // Tovid likes to dump in the cwd. Chdir so that tovid dumps its temp files somewhere usefulish

define('MAXVOB',1073709056); // The max size of a single VOB
global $pids;
$pids = Array();

$currentdvdsize = MENUOVERHEAD;
$currentdvdfiles = Array();
fgetcsv($fh); // remove header from csv file
while($file = fgetcsv($fh)){

    // Fast check!
    if((filesize(INPUTDIR .'/'. $file[0]) + $currentdvdsize) > DVDSIZE){
    makeDVD($currentdvdfiles); // Make a DVD!

    $currentdvdfiles = Array(); // Reset!
    $currentdvdsize = MENUOVERHEAD;
    }

    $currentdvdfiles[] = $file;

    // Calculate how much space this video will take on the disc
    // dvdauthor seems to:
    // 1) Never mix videos in the same VOB
    // 2) Make all VOBs except the last one come out to MAXVOB size (the last one can be whatever size smaller than MAXVOB)
    $currentdvdsize += (MAXVOB * ceil(filesize(INPUTDIR .'/'. $file[0])/MAXVOB));

    if($currentdvdsize > DVDSIZE){
    die("It looks like {$file[0]} is too big to fit on a DVD!\n");
    }
}

// Make any remnants
if(count($currentdvdfiles) > 0){
    makeDVD($currentdvdfiles);
}

function makeDVD($files){
    global $pids;

    // Make title
    $last = count($files) - 1;
    if($files[0][1] != $files[$last][1]){
    $title = "{$files[0][1]} - {$files[$last][1]}";
    }else{
    $title = "{$files[0][1]}";
    }

    $count = 2;
    $origtitle = $title;
    while(file_exists(OUTDIR . "/$title")){
    $title = "$origtitle disc $count";
    $count++;
    }

    // List of files
    $input = Array();
    $shorttitles = Array();
    $fulltitles = Array();
    foreach($files as $file){
    $input[] = INPUTDIR . "/{$file[0]}";
    $fulltitles[] = $file[2];
    $shorttitles[] = $file[3];
    }

    $cmd = "tovid disc
    -files  " . implode(" ",array_map('escapeshellarg',$input)) . "
    -titles " . implode(' ',array_map('escapeshellarg',$shorttitles)) . "
    -menu-title " . escapeshellarg($title) . "
    -menu-fontsize 18
    -title-color '#ff7700'
    -title-stroke black
    -titles-fontsize 18
    -titles-color '#ff7700'
    -showcase-titles-align east
    -rotate 5
    -wave default
    -submenus
    -submenu-titles " . implode(' ',array_map('escapeshellarg',$fulltitles)) . "
    -submenu-title-color '#ff7700'
    -submenu-stroke black
    -loop 0
    -submenu-length 20
    -noask
    -out " . escapeshellarg(OUTDIR . "/$title");

    // Now wait for our turn...
    while(count($pids) >= MAXCHILDREN){
    $status = NULL;
    $exited_pid = pcntl_wait($status);            
    if(pcntl_wexitstatus($status) != 0){
        print "FAILURE IN " . TMPDIR . "/tovid.$exited_pid!!!\n{$pids[$exited_pid]}\n";
    }
    unset($pids[$exited_pid]);
    }

    $cmd = str_replace("\n",' ',$cmd);

    print "LAUNCHING!!!\n$cmd\n";

    $pid = pcntl_fork();
    if($pid == -1){
    die("COULDNT FORK!");
    }else if($pid){
    $pids[$pid] = $cmd;
    sleep(3); // give tovid a chance to claim its temp directories
    }else{
    $cmd .= " > " . TMPDIR . "/tovid." . getmypid() . " 2>&1";
    exec($cmd);
    exit();
    }
}

Using the Script

Save the PHP script above to a file named tovidBatch.php and make it executable.

Edit the defined constants to fit your directories and output media (DVDSIZE). If desired, edit the tovid disc command to build your DVDs the way you want them.

Finally, run (on a command line)

php tovidBatch.php

Your videos are on their way!

It has been a big disapointment and I do not plan on buying an HP laptop again and can’t recommend that anyone else buy one either.

I chose the dv7 because I needed something with some processing power. My dv7 has anIntel i7 clocked at 1.87Ghz, with 6 gigs of RAM.

Update (January, 23, 2011)

The screw that was loose on the case eventually came out. Within a day the hinge near where the screw fell out of broke, and I could hear a piece of metal shaking around inside the case. The warranty expires at the end of March.

The HP support person I spoke with was professional in the way that a police officer who pulls you over for speeding is professional. He kept trying to get me to say that I had dropped, bumped or otherwise abused the laptop and warned me multiple times that if I sent it in and they found signs of abuse that I would have to pay for repairs.

Still, 50 minutes later and a box was on its way to my house. The box took 2 days to get here, 2 days to get to HP, and they must’ve fixed it that day because I got it back 3 days after that.

I had only complained about the hinge. The loose power plug and missing keyboard/worn off keyboard I had chalked up to wear-and-tear and didn’t expect them to fix under warranty, but they did.

So, do I love HP laptops now? No, but I’m happy with how their warranty service turned out, and hopeful that I won’t have these problems once the warranty expires!

The HP Pavilion dv7 Overheats

Unfortunately, the dv7 is inadequately cooled. When I start the computer temperature of the computer quicky reaches 75-78 degress (Celcius) and the fans are running loudly. As soon as I start a couple of desktop applications (Firefox, Pidgin, Thunderbird and a terminal) the temperature reaches about 85.

When I got this machine, I was excited because from time to time I rip VHS tapes (old homevideos) and convert them to DVDs for family members. The encoding process is long and slow, and I expected to benefit from all 4 cores.

Sadly, I can’t even run a video encoder at full speed on a single core without the computer overheating, and shutting down.  The computer shuts down if it reaches 100 degrees. I have to run ffmpeg (or mencoder) at 55% CPU usage (of one core) in oder for the fans to keep up with the CPU.

I’m effectivly stuck with a single core 1.03 GHz (55% of a single 1.87GHz core) computer since I can’t use more than that at once. Extremely disapointing.

TIP: If you're in a similar situation, and you're on Linux, you
can use the program cpulimit throttle a specific process or
program. When I try to encode video on Windows (in iTunes) the
computer overheats no matter what I do.

NOTE #1:  This happens in both Windows AND Linux, so it’s not some weird Linux bug. I nearly always use my laptop on a hard surface — and I have to have it on something hard with great airflow if I want to even think about encoding a video.

NOTE #2: My fans and vents are clean. I can see through them to the fans, and have used compressed air on them anyways just in case.

The HP Pavilion dv7 Has a Low Quality Keyboard

I admit that I use  my computer more than most people do. Between work and hobbies, it’s not unusual for me to be on my computer 10-12 hours a day, and often more.

This laptop however, is the first one that I have ever worn the letters off the keys. It is also the first one that I have ever had a key spring break during normal use.

Luckly the dv7 comes with a numeric keypad which I don’t use, so I took the keyspring from one of those keys and repaired my broken key.

Again, very disapointing

The HP Pavilion dv7 Case is Poorly Built

Within a few months of getting my dv7 I noticed that the screen wouldn’t stay in position. It would always settle back an extra inch from where I had pushed it. It turned out that the top plastic piece of the case was coming loose from the bottom piece, which in turn left the lid hinge a little loose.

I tightened the screw on the case, and have had to tighten it again every couple of months.

I’m Done Whining Now

I don’t want/need a replacement from HP badly enough to talk to someone in a call center. I’m just going to keep using this computer until it dies. I’m betting it doesn’t make it a year. My guess is that the threads for the case screw eventually wear out and the screen won’t stay up.

I have a rock-solid reliable P4 desktop computer (an HP, no less!) I can use if this thing goes, and next time I’ll buy something more reliable, like a Thinkpad or a Mac Book.

So, if you’re in the market for a laptop, use extreme caution when buying an HP Pavilion. You might not get as good of a machine as you expect.

It also gave me a single place to put a try/catch block with a debug_print_backtrace to figure out where the offending query was coming from. In my admitedly brief experience with MongoDB so far, Mongo doesn’t lend itself to such practices quite as easily.

With this project I’ve been working on at work we’ve got 34,000 lines of code. We have a global $mongo object that we run queries against. When a Mongo query chokes up our site I need to know what exact call has been made, with what arguments and coming from where.

So I wrote this shim, MongoLogger that would pretends to be Mongo. During development I replace the $mongo object with my MongoLogger object, like so:

$mongodb = new Mongo("mongodb://un:pw@host/db"); // Connect to Mongo
$mongo = $mongodb->db;                   // pick your database
$mongo = new MongoLogger($mongo); // Replace the $mongo

MongoLogger then uses the PHP magic methods __call and __get and the almost magic method call_user_func_array to do the needed logging, and to pass the arguments on to the real $mongo object.

Note: Magic Methods, an call_user_func_array in particular, are slow. Not recommended for production use!

This is simpler than what I wrote for work, but the concept is the same and the important bits are here. It’s what I wish I could’ve found when I first started worrying about how to debug MongoDB calls.

<?php

class MongoLogger{
    /**
     * @class MongoLogger
     *
     * @brief A debugging shim between your code and MongoDB. Should be disbaled for production
     *
     * Usage:
     * $mongodb = new Mongo("mongodb://un:pw@host/db"); // Connect to Mongo
     * $mongo = $mongodb->db;                           // pick your database
     * $mongo = new MongoLogger($mongo);                // and all calls made to the original $mongo just keep working...
     *
     * YAY MAGIC METHODS! http://php.net/manual/en/language.oop5.magic.php
     * Makes use of magic methods __construct, __call and __get and the almost magic method call_user_func_array.
     */

    var $collections = Array();

    function __construct($mongo){
    global $real_mongo;
    $real_mongo = $mongo;
    }

    public function __call($name,$args){
    global $real_mongo;

    error_log("Mongo->$name called with " . print_r($args,TRUE));

    try {
        $res = call_user_func_array(Array($real_mongo,$name),$args);
        return $res;
    } catch (Exception $e){
        debug_print_backtrace();
        error_log(print_r($e,TRUE));
        throw $e;
    }
    }

    /**
     * @brief Return a MongoLoggerCollection for the requested collection
     * This is the main function that gets called in this class. We just pretend that
     * we've got whatever we're asked for (just like Mongo does)
     *
     */
    public function __get($name){
    if(!array_key_exists($name,$this->collections)){ $this->collections[$name] = new MongoLoggerCollection($name); }
    return $this->collections[$name];
    }
}

class MongoLoggerCollection {
    /**
     * @class MongoLoggerCollection
     *
     * @brief Represents a collection in Mongo. Most of your calls will come through here.
     */
    var $collection;

    function __construct($name){
    $this->collection = $name;
    }

    public function __call($name,$args){
    global $real_mongo;

    // Logging!
    $debug = debug_backtrace();
    $caller = $debug[1];
    error_log($caller['file'] . ':' . $caller['line']);
    error_log("\$mongo->{$this->collection}->$name(");
    foreach($args as $arg){ error_log(print_r($arg,TRUE)); }
    error_log(")");

    try {
        $res = call_user_func_array(Array($real_mongo->{$this->collection},$name),$args);
        return $res;
    } catch (Exception $e){
        debug_print_backtrace();
        error_log(print_r($e,TRUE));
        throw $e;
    }
    }
}