Re: [greenstone-users] IPTC and RDF metadata

From Stephen DeGabrielle
DateFri, 16 Sep 2005 01:01:28 +0930
Subject Re: [greenstone-users] IPTC and RDF metadata
In-Reply-To (IMTBWQ$ADAB7CC681412E1BF569605E0F3FD01B-scarlet-be)
the identify command that is included in imagemagick will work.

I just tried it on;
http://jigsaw.w3.org/Yves/Australia/1998/04/Brisbane-06.jpg

and got the following at the command line:

stephen:~ spdegabrielle$ identify -verbose /Users/spdegabrielle/bird.jpg       /Users/spdegabrielle/bird.jpg JPEG 792x1216 DirectClass 253kb 0.180u 0:01
Image: /Users/spdegabrielle/bird.jpg
  Format: JPEG (Joint Photographic Experts Group JFIF format)
  Geometry: 792x1216
  Class: DirectClass
  Colorspace: RGB
  Type: TrueColor
  Depth: 8 bits
  Endianess: Undefined
  Channel depth:
    Red: 8-bits
    Green: 8-bits
    Blue: 8-bits
  Channel statistics:
    Red:
      Min: 1
      Max: 255
      Mean: 70.3238
      Standard deviation: 61.367
    Green:
      Min: 0
      Max: 249
      Mean: 58.2279
      Standard deviation: 49.8561
    Blue:
      Min: 0
      Max: 253
      Mean: 45.5241
      Standard deviation: 40.4305
  Colors: 147310
  Rendering-intent: Undefined
  Resolution: 900x900
  Units: PixelsPerInch
  Filesize: 253kb
  Interlace: Plane
  Background Color: white
  Border Color: #DFDFDF
  Matte Color: grey74
  Dispose: Undefined
  Iterations: 0
  Compression: JPEG
  Quality: 94
  Orientation: Undefined
  Comment:
<?xml version="1.0"?>
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns# "
    xmlns="http://www.w3.org/1999/09/28-Photo-ns#"
     xmlns:DC="http://purl.oclc.org/dc/documents/rec-dces-199809.htm# "
     xmlns:Technical="http://www.w3.org/2000/PhotoRDF/technical-1-0#">
  <rdf:Description about="">
    <DC:Description>A nice, colorful bird near Mt Tambourine, Australia. It has a nice blue tail, read and blue wings and head, it's looking at you!</DC:Description>
    <DC:Type>image/jpeg</DC:Type>
    <DC:Subject>Landscape</DC:Subject>
    <DC:Source>35mm slide</DC:Source>
    <DC:Rights> http://www.w3.org/People/Lafon/Copyright?1998</DC:Rights>
    <DC:Title>Bird near Mt Tambourine</DC:Title>
    <DC:Date>1998-04-20</DC:Date>
    <DC:Coverage>Brisbane - Australia</DC:Coverage>
    <DC:Creator>http://www.w3.org/People/Lafon/</DC:Creator>
    <Technical:devel-date>2000-05-01</Technical:devel-date>
    <Technical:lens>Minolta AF70-210</Technical:lens>
    <Technical:camera>http://www.w3.org/People/Lafon/Minolta800i</Technical:camera>
    <Technical:film>Fuji</Technical:film>
  </rdf:Description>
</rdf:RDF>
  JPEG-Colorspace: 2
  JPEG-Sampling-factors: 1x1,1x1,1x1
  Signature: 3006c32a0d01052a9fe1e980e76eaa74f7df2a6633f5809d6992f3d618b62e86
  Profile-iptc: 3788 bytes
    unknown: 0x00000000: ff4a6446 556bff09 ff0671ff ff46ff00           --JdFUk-
  Profile-icc: 3144 bytes
  Tainted: False
  User Time: 0.180u
  Elapsed Time: 0:01
  Version: ImageMagick 6.1.7 12/19/04 Q16 http://www.imagemagick.org
stephen:~ spdegabrielle$

It should be simple enough to modify to meet your needs.

This is a version of imagePlug that I did in a couple of minutes.

It just uses identify -verbose and a regular expression to get the comment data out.

[sorry it's messy & uncommented]

A few more regular expressions could be used to get the metadata into seperate fields as required.
(send me a sample image and I may be able to hack something together over the weekend)

s.

###########################################################################
#
# ImagePlug.pm -- simple text plugin
# A component of the Greenstone digital library software
# from the New Zealand Digital Library Project at the
# University of Waikato, New Zealand.
#
# Copyright (C) 1999 New Zealand Digital Library Project
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
#
###########################################################################

package ImagePlug;

use BasPlug;

use strict;
no strict 'refs'; # allow filehandles to be variables and viceversa

sub BEGIN {
    @ImagePlug::ISA = ('BasPlug');
}

my $arguments =
    [ { 'name' => "process_exp",
    'desc' => "{BasPlug.process_exp}",
    'type' => "regexp",
    'deft' => &get_default_process_exp(),
    'reqd' => "no" },
      { 'name' => "noscaleup",
    'desc' => "{ImagePlug.noscaleup}",
    'type' => "flag",
    'reqd' => "no" },
      { 'name' => "thumbnailsize",
    'desc' => "{ImagePlug.thumbnailsize}",
    'type' => "int",
    'deft' => "100",
    'range' => "1,",
    'reqd' => "no" },
      { 'name' => "thumbnailtype",
    'desc' => "{ImagePlug.thumbnailtype}",
    'type' => "string",
    'deft' => "gif",
    'reqd' => "no" },
      { 'name' => "screenviewsize",
    'desc' => "{ImagePlug.screenviewsize}",
    'type' => "int",
    'deft' => "0",
    'range' => "1,",
    'reqd' => "no" },
      { 'name' => "screenviewtype",
    'desc' => "{ImagePlug.screenviewtype}",
    'type' => "string",
    'deft' => "jpg",
    'reqd' => "no" },
      { 'name' => "converttotype",
    'desc' => "{ImagePlug.converttotype}",
    'type' => "string",
    'deft' => "",
    'reqd' => "no" },
      { 'name' => "minimumsize",
    'desc' => "{ImagePlug.minimumsize}",
    'type' => "int",
    'deft' => "100",
    'range' => "1,",
    'reqd' => "no" } ];

my $options = { 'name'     => "ImagePlug",
        'desc'     => "{ ImagePlug.desc}",
        'abstract' => "no",
        'inherits' => "yes",
        'args'     => $arguments };



sub new {
    my ($class) = shift (@_);
    my ($pluginlist,$inputargs,$hashArgOptLists) = @_;
    push(@$pluginlist, $class);

    if(defined $arguments){ push(@{$hashArgOptLists->{"ArgList"}},@{$arguments});}
    if(defined $options) { push(@{$hashArgOptLists->{"OptList"}},$options)};

    my $self = (defined $hashArgOptLists)? new BasPlug($pluginlist,$inputargs,$hashArgOptLists): new BasPlug($pluginlist,$inputargs);

    # Check that ImageMagick is installed and available on the path (except for Windows 95/98)
    if (!($ENV{'GSDLOS'} eq "windows" && !Win32::IsWinNT())) {
    my $result = `identify 2>&1`;
    if ($? == -1 || $? == 256) {  # Linux and Windows return different values for "program not found"
        $self->{'imagemagick_not_installed'} = 1;
    }
    }

    return bless $self, $class;
}

sub get_default_process_exp {
    my $self = shift (@_);

    return q^(?i)(.jpe?g|.gif|.png|.bmp|.xbm|.tif?f)$^;
}

# this makes no sense for images
sub block_cover_image
{
    my $self =shift (@_);
    my ($filename) = @_;

    return;
}
# Create the thumbnail and screenview images, and discover the Image's
# size, width, and height using the convert utility.

sub run_convert {
    my $self = shift (@_);
    my $filename = shift (@_);   # filename with full path
    my $file = shift (@_);       # filename without path
    my $doc_obj = shift (@_);
    my $section = $doc_obj->get_top_section();
   
    my $verbosity = $self->{'verbosity'};
    my $outhandle = $self->{'outhandle'};

    # check the filename is okay
    return 0 if ($file eq "" || $filename eq "");

#    Code now extended to quote filenames in 'convert' commnads
#    Allows spaces in filenames, but note needs spaces to be escaped in URL as well
#    if ($filename =~ m/ /) {
#    print $outhandle "ImagePlug: "$filename" contains a space. choking. ";
#    return undef;
#    }

    my $minimumsize = $self->{'minimumsize'};
    if (defined $minimumsize && (-s $filename < $minimumsize)) {
        print $outhandle "ImagePlug: "$filename" too small, skipping "
        if ($verbosity > 1);
    }


    # Convert the image to a new type (if required).
    my $converttotype = $self->{'converttotype'};
    my $originalfilename = "";  # only set if we do a conversion
    my $type = "unknown";


    if ($converttotype ne "" && $filename =~ m/$converttotype$/) {

    $originalfilename = $filename;
    $filename = &util::get_tmp_filename() . ".$converttotype";
    $self->{'tmp_filename'} = $filename;

    my $command = "convert -interlace plane -verbose "$originalfilename" "$filename"";
    print $outhandle "$command " if ($verbosity > 2);
    my $result = '';
    $result = `$command`;
    print $outhandle "RESULT = $result " if ($verbosity > 2);

    $type = $converttotype;
    }
   

    # Add the image metadata
    my $url = $file;
    $url =~ s/ /%20/g;

    $doc_obj->add_metadata ($section, "Image", $url);

    # Also want to set filename as 'Source' metadata to be
    # consistent with other plugins
    $doc_obj->add_metadata ($section, "Source", $url);

    my ($image_type, $image_width, $image_height, $image_size, $image_comments)
    = &identify($filename, $outhandle, $verbosity);
    print $outhandle "COMMENTS = $image_comments " if ($verbosity > 2);
   
    if ($image_type ne " ") {
    $type = $image_type;
    }
   
    $doc_obj->add_metadata ($section, "FileFormat", $type);
    $doc_obj->add_metadata ($section, "FileSize",   $image_size);

    $doc_obj->add_metadata ($section, "ImageType",   $image_type);
    $doc_obj->add_metadata ($section, "ImageWidth",  $image_width);
    $doc_obj->add_metadata ($section, "ImageHeight", $image_height);
    $doc_obj->add_metadata ($section, "ImageSize",   $image_size);
    $doc_obj->add_metadata ($section, "ImageComments", $image_comments);
$doc_obj->add_text($section, $image_comments);
    $doc_obj->add_metadata ($section, "srclink",
                "<a href="/gsdl/collect/gsarch/index/assoc/[assocfilepath]/[Image]">");
    $doc_obj->add_metadata ($section, "/srclink", "</a>");

    $doc_obj->add_metadata ($section, "srcicon", "<img src="/gsdl/collect/gsarch/index/assoc/[assocfilepath]/[Image]" width=100>");

   
    # Add the image as an associated file
    $doc_obj->associate_file($filename,$file,"image/$type",$section);


    # Make the thumbnail image
    my $thumbnailsize = $self->{'thumbnailsize'} || 100;
    my $thumbnailtype = $self->{'thumbnailtype'} || 'gif';
   
    my $thumbnailfile = &util::get_tmp_filename() . ".$thumbnailtype";
    $self->{'tmp_filename2'} = $thumbnailfile;

    # Generate the thumbnail with convert
    my $command = "convert -interlace plane -verbose -geometry $thumbnailsize"
    . "x$thumbnailsize "$filename" "$thumbnailfile"";
    print $outhandle "THUMBNAIL: $command " if ($verbosity > 2);
    my $result = '';
    $result = `$command 2>&1` ;
    print $outhandle "THUMB RESULT: $result " if ($verbosity > 2);

    # Add the thumbnail as an associated file ...
    if (-e "$thumbnailfile") {
    $doc_obj->associate_file("$thumbnailfile", "thumbnail.$thumbnailtype",
                 "image/$thumbnailtype",$section);
    $doc_obj->add_metadata ($section, "ThumbType", $thumbnailtype);
    $doc_obj->add_metadata ($section, "Thumb", "thumbnail.$thumbnailtype");

    $doc_obj->add_metadata ($section, "thumbicon", "<img src="/gsdl/collect/gsarch/index/assoc/[assocfilepath]/[Thumb]" width=[ThumbWidth] height=[ThumbHeight]>");
    }

    # Extract Thumnail metadata from convert output
    if ($result =~ m/[0-9]+x[0-9]+=>([0-9]+)x([0-9]+)/) {
    $doc_obj->add_metadata ($section, "ThumbWidth", $1);
    $doc_obj->add_metadata ($section, "ThumbHeight", $2);
    }

    # Make a screen-sized version of the picture if requested
    if ($self->{'screenviewsize'}) {

    # To do: if the actual image smaller than the screenview size,
    # we should use the original !

    my $screenviewsize = $self->{'screenviewsize'};
    my $screenviewtype = $self->{'screenviewtype'} || 'jpeg';
    my $screenviewfilename = &util::get_tmp_filename() . ".$screenviewtype";
    $self->{'tmp_filename3'} = $screenviewfilename;

    # make the screenview image
    my $command = "convert -interlace plane -verbose -geometry $screenviewsize"
        . "x$screenviewsize "$filename" "$screenviewfilename"";
    print $outhandle "$command " if ($verbosity > 2);
    my $result = "";
    $result = `$command  2>&1` ;
    print $outhandle "$result " if ($verbosity > 3);

    # get screenview dimensions, size and type
        if ($result =~ m/[0-9]+x[0-9]+=>([0-9]+)x([0-9]+)/) {
        $doc_obj->add_metadata ($section, "ScreenWidth", $1);
        $doc_obj->add_metadata ($section, "ScreenHeight", $2);
    }
    else {
        $doc_obj->add_metadata ($section, "ScreenWidth", $image_width);
        $doc_obj->add_metadata ($section, "ScreenHeight", $image_height);
    }

    #add the screenview as an associated file ...
    if (-e "$screenviewfilename") {
        $doc_obj->associate_file("$screenviewfilename", "screenview.$screenviewtype",
                     "image/$screenviewtype",$section);
        $doc_obj->add_metadata ($section, "ScreenType", $screenviewtype);
        $doc_obj->add_metadata ($section, "Screen", "screenview.$screenviewtype");

        $doc_obj->add_metadata ($section, "screenicon", "<img src="/gsdl/collect/gsarch/index/assoc/[assocfilepath]/[Screen]" width=[ScreenWidth] height=[ScreenHeight]>");
    } else {
        print $outhandle "ImagePlug: couldn't find "$screenviewfilename" ";
    }
    }

    return $type;


}



# Discover the characteristics of an image file with the ImageMagick
# "identify" command.

sub identify {
    my ($image, $outhandle, $verbosity) = @_;

    # Use the ImageMagick "identify" command to get the file specs
    my $command = "identify "$image" 2>&1";
    print $outhandle "$command " if ($verbosity > 2);
    my $result = '';
    $result = `$command`;
    print $outhandle "$result " if ($verbosity > 3);

    # Use the ImageMagick "identify" command to get the file specs
    my $command2 = "identify -verbose "$image" 2>&1";
    print $outhandle "$command2 " if ($verbosity > 2);
    my $result2 = '';
    $result2 = `$command2`;
  #  print $outhandle "XX$result2 " if ($verbosity > 3);

    # Read the type, width, and height
    my $type =   'unknown';
    my $width =  'unknown';
    my $height = 'unknown';
    my $comment = 'unknown';
    my $image_safe = quotemeta $image;
    if ($result =~ /^$image_safe (w+) (d+)x(d+)/) {
    $type = $1;
    $width = $2;
    $height = $3;
    }

    if ($result2 =~ /.*Comment:(.*)JPEG-Colorspace.*/s) {
    $comment = $1;
    }
    print $outhandle "XX2XX $result2 " if ($verbosity > 3);
    # Read the size
    my $size = "unknown";
    if ($result =~ m/^.* ([0-9]+)b/) {
    $size = $1;
    }
    elsif ($result =~ m/^.* ([0-9]+)(.([0-9]+))?kb?/) {
    $size = 1024 * $1;
    if (defined($2)) {
        $size = $size + (1024 * $2);
        # Truncate size (it isn't going to be very accurate anyway)
        $size = int($size);
    }
    }

    print $outhandle "file: $image: $type, $width, $height, $size "
    if ($verbosity > 2);

    # Return the specs
    return ($type, $width, $height, $size, $comment);
}


# The ImagePlug read() function. This function does all the right things
# to make general options work for a given plugin. It calls the process()
# function which does all the work specific to a plugin (like the old
# read functions used to do). Most plugins should define their own
# process() function and let this read() function keep control.
#
# ImagePlug overrides read() because there is no need to read the actual
# text of the file in, because the contents of the file is not text...
#
# Return number of files processed, undef if can't process
# Note that $base_dir might be "" and that $file might
# include directories

sub read {
    my $self = shift (@_);
    my ($pluginfo, $base_dir, $file, $metadata, $processor, $maxdocs, $total_count, $gli) = @_;

    my $outhandle = $self->{'outhandle'};

    my $filename = &util::filename_cat($base_dir, $file);
    return 0 if $self->{'block_exp'} ne "" && $filename =~ /$self->{'block_exp'}/;
    if ($filename !~ /$self->{'process_exp'}/ || !-f $filename) {
    return undef;
    }

    print STDERR "<Processing n='$file' p='ImagePlug'> " if ($gli);
    print $outhandle "ImagePlug processing $file "
        if $self->{'verbosity'} > 1;

    # None of this works very well on Windows 95/98...
    if ($ENV{'GSDLOS'} eq "windows" && !Win32::IsWinNT()) {
    if ($gli) {
        print STDERR "<ProcessingError n='$file' r='Windows 95/98 not supported'> ";
    }
    print $outhandle "ImagePlug: Windows 95/98 not supported ";
    return -1;
    }

    # None of this is going to work very well without ImageMagick...
    if ($self->{'imagemagick_not_installed'}) {
    if ($gli) {
        print STDERR "<ProcessingError n='$file' r='ImageMagick not installed'> ";
    }
    print $outhandle "ImagePlug: ImageMagick not installed ";
    return -1;
    }

    #if there's a leading directory name, eat it...
    $file =~ s/^.*[/\]//;
   
    # create a new document
    my $doc_obj = new doc ($filename, "indexed_doc");
    $doc_obj->set_OIDtype ($processor->{'OIDtype'});   
    $doc_obj->add_utf8_metadata($doc_obj->get_top_section(), "Plugin", "$self->{'plugin_type'}");

    #run convert to get the thumbnail and extract size and type info
    my $result = run_convert($self, $filename, $file, $doc_obj);
   
    if (!defined $result)
    {
    if ($gli) {
        print STDERR "<ProcessingError n='$file'> ";
    }
    print $outhandle "ImagePlug: couldn't process "$filename" ";
    return -1; # error during processing
    }

    #create an empty text string so we don't break downstream plugins
    my $text = &gsprintf::lookup_string("{BasPlug.dummy_text}");
    # include any metadata passed in from previous plugins
    # note that this metadata is associated with the top level section
    my $section = $doc_obj->get_top_section();
    $self->extra_metadata ($doc_obj, $section, $metadata);

    # do plugin specific processing of doc_obj
    unless (defined ($self->process($text, $pluginfo, $base_dir, $file, $metadata, $doc_obj))) {
    print STDERR "<ProcessingError n='$file'> " if ($gli);
    return -1;
    }

    # do any automatic metadata extraction
    $self->auto_extract_metadata ($doc_obj);

    # if we haven't found any Title so far, assign one
    # this was shifted to here from inside read()
    $self->title_fallback($doc_obj,$section,$file);
    # add an OID
    $doc_obj->set_OID();
    #$doc_obj->add_text($section, $text);

    # process the document
    $processor->process($doc_obj);

    # clean up temporary files - we do this here instead of in 
    # run_convert because associated files aren't actually copied
    # until after process has been run.
    if (defined $self->{'tmp_filename'} &&
    -e $self->{'tmp_filename'}) {
      &util::rm($self->{'tmp_filename'})
    }
    if (defined $self->{'tmp_filename2'} &&
    -e $self->{'tmp_filename2'}) {
      &util::rm($self->{'tmp_filename2'})
    }
    if (defined $self->{'tmp_filename3'} &&
    -e $self->{'tmp_filename3'}) {
      &util::rm($self->{'tmp_filename3'})
    }

    $self->{'num_processed'}++;

    return 1;
}

# do plugin specific processing of doc_obj
sub process {
    my $self = shift (@_);
    my ($textref, $pluginfo, $base_dir, $file, $metadata, $doc_obj) = @_;
    my $outhandle = $self->{'outhandle'};
   
    return 1;
}

1;


#----
#Stephen De Gabrielle
#