2010/11/15-18,22-23 - Happy Turkey Day!

It has been a good couple weeks. I've managed to stay out of trouble, and everyone is looking forward to Thanksgiving. I also managed to get a few things done. First, we imported Claudine's pamphlet collection into our DSpace repository. Jon installed the Solaris freeware xpdf package on repo, which allowed me to enable the dspace support for pdf-thumbnails, and Tony helped fix the layout problems that popped up when the thumbnails came online. I also imported the collection metadata into the vufind index on devcat with our vufind-OAI harvest tool and this xsl.

We interviewed a couple good candidates for the systems department position. The poor candidates each endured three hours of meetings, and we interviewed two candidates in a day, so it was exhausting for everyone. Two more candidates to go next week.

I checked in a couple vufind patches. One patch adds a link to a record's holdings screen that auto-populates the AUBIExpress (illiad) request form. The other patch fixes the call-number refinement in the search facets that I broke a while ago replacing AJAX facets. I think Clint will probably get the ok to release the patches next week.

Finally, I was able to setup authentication with dspace credentials for the minnows data-submission webapp. This bLog post has more details.

2010/11/8-11 - whoops!

This week had good news and bad news - bad news first.

There was a little excitement this week as we accidentally released a bug to our live vufind server. On Monday we flipped the switch on a VM Clint had prepared for release that included a bug (that I wrote - ugh!) in the AJAX handler that marries a call-number to its proper item in a search result. The bug had gone unnoticed on our test server for over a week, and didn't lead to serious problems till Monday night, when search results started popping up with wacky call numbers. The reference desk paged systems, but the page went unnoticed till Tuesday morning (before I came in), when Denise and Jon called Clint at home on vacation. Clint quickly found the bad commit I had checked in, and reverted the bug back out. I keep expecting someone to throw a fit, but people have been cool, so maybe I'm too cynical.

The good news was that Claudine signed off on the metadata massaging that my import program runs through to marry her collection of scans with catalog MARC records for import into d-space. This project has been lingering since June - hopefully the end is in sight. I'll import the rest of the collection on Monday, and setup some initial XSL for Marliese to index the collection into vufind on devcat.

Finally, the vufind bug inspired me to setup some vufind regressions. The following e-mail excerpt explains what I did.

--- Reuben Pasquini 11/13/10 8:52 PM ---
Hi Clint!

I setup a framework for vufind regression tests that we can
add to over time.
    http://code.google.com/p/littleware/source/browse/?repo=catalog#hg/VufindRegression/src/main/scala/edu/auburn/library/tool/vufindTest

You can download a build of the test runner at:
    http://devcat.lib.auburn.edu/tests/vufindTest.zip
- just run 'vufindTest.bat'.

The test suite currently just runs through 3 sets of tests against devcat.
The first test runs a "Harry Potter" search (
    using HtmlUnit http://htmlunit.sourceforge.net/
), and verifies that the results include a couple expected records,
and that the call-numbers are correct on those 2 records.

The second test loads a specific Harry Potter record, then
checks the title,
verifies that the record has some holdings information,
and checks that the "campus delivery" cookies are set.

Finally, the third test accesses
    http://devcat.lib.auburn.edu/vufind/AuburnTestSuite.php
which runs through a suite of PHP unit tests
    http://devcat.lib.auburn.edu/cgi-bin/hgwebdir.cgi/vufind/file/tip/web/AuburnTestSuite.php

Anyway, we can start running these regressions before releasing new code.
I can set you up with a copy of the code if you want to add some tests.
I want to add tests that directly access some of the AJAX services, 
add a test fixture for Solr.php,
and setup a Hudson server (http://hudson-ci.org/) to
automatically run the tests whenever we commit some code to the vufind repository,
but I probably won't get to that for a while ...

Let me know if you run into problems with the test runner ...

Cheers,
Reuben

2010/11/1-4 - election day!

It was another quiet week at the library. I spent some time updating the littleware build configs to simplify setting up the build for a new project. The build.xml files for each project now share a master build-template that defines some common cross-project build rules, and automatically installs ivy. I also updated the ivy.xml files to define compile, client, server, ee_client, ee_server, and test standard configurations; and to reference latest.integration dependency versions where appropriate. I hope to setup a hudson continuous integration server in the next few months to automate the test, artifact release, and server deploy processes.

I put most of the code in place that integrates the Google spread sheet data with the ACES collection d-space import tool. I hope to have a version of the collection on the server next week.

Finally, I helped with a couple small vufind bugs. Clint worked hard this week to release a few patches to the production server. There's some kind of big "vufind, love it or hate it" meeting next week. There are plenty of flaws with our vufind setup, but it has a lot of advantages over our only alternative.

2010/10/25-28 - Happy Halloween

I got a few nice things working this week. First, the multi-step minnows submission process is running. I scheduled a meeting for Thursday, but Jon forgot about it, so we'll try again next week.

Next, I integrated Barbara Bishop's gifs of the library maps into our devcat vufind install. I used ImageMagic to down-rez Barbara's images, and convert them to PNG, and wired up vufind to popup a YUI-2 panel to display the map. Barbara and Marliese gave some good feedback, but we haven't released the code to catalog yet - bla.

When working on the maps thing I stumbled across the fact that our "retrieve X from stacks" request system consists of a patron writing down what he/she wants and handing it to circulation. I'd like to try to setup an online system - submit a request, and get an e-mail or text when the item is at circulation kind of thing.

Finally I implemented a MultiVoyager.php vufind driver for some people on the vufind e-mail list. I sent out the following e-mail, but haven't heard back from anyone. Bla.

--- Reuben Pasquini 10/31/2010 4:57 PM ---
Hi David,

I was able to get a simple 2-ILS system working at Auburn on our dev server.
Here's what I did.

*. Setup a solrmarc import that prepends an ILS-specific prefix to each record.
    You can get something working quickly by modifying the 'id' rule in
    the solrmarc marc.properties to:

      id = script(customId.bsh),getCustomId("PREFIX")

    where customId.bsh has:

import org.marc4j.marc.Record;
import org.marc4j.marc.ControlField;


/**
 * Generate a custom id as prefix + 001
 */
public String getCustomId( Record record, String prefix ) {
    return prefix + ((ControlField) record.getVariableField( "001" )).getData();
}

.........................................

*. I wrote a 
      MultiVoyager.php
driver that reads
     MultiVoyager.ini
and maintains a list of Voyager.php driver instances.
I attached a copy of MultiVoyager.ini, or
you can clone a copy out of our Mercurial repository:
     http://catalog.lib.auburn.edu/cgi-bin/hgwebdir.cgi/vufind/ 

The MultiVoyager.ini file maps a prefix to and ILS config -
an example follows.
The "EMPTY" keyword indicates no prefix.
This example works for records that look like
     12345 (no prefix)
and 
     AUM12345 (AUM prefix).

MultiVoyager.ini:

[Catalog]
configList = EMPTY,AUM
defaultDriver = EMPTY
[EMPTY]
host        = bla
port        = 1521
service     = VGER
user        = bla
password    = bla
database    = bla
pwebrecon   = http://pooh.lib.auburn.edu:7008/vwebv/holdingsInfo 
[AUM]
host        = bla
port        = 1521
service     = VGER
user        = bla
password    = bla
database    = bla
pwebrecon   = http://pooh.lib.auburn.edu:7008/vwebv/holdingsInfo 

........................................

*. I had to patch a few other php lines here and there too
    to get the ajax load of holding status working right,
    modify Voyager.php to accept a config in the constructor
    rather than load Voyager.ini, ... - stuff like that.
    The complete patch is here, but our vufind code is
    pretty far out of date compard to vufind.org:
       http://catalog.lib.auburn.edu/cgi-bin/hgwebdir.cgi/vufind/rev/9e94166d5e0b 


........................

Hope this code works for you - let me know how it goes.

Cheers,
Reuben


-----Original Message-----
From: Reuben Pasquini [mailto:rdp0004@auburn.edu] 
Sent: Monday, October 18, 2010 1:20 PM
To: Harmon, Kelly; vufind-general@lists.sourceforge.net; Osullivan L.
Subject: Re: [VuFind-General] Anyone merging more than one
VoyagerDatabaseintoVuFind?

At Auburn we currently only index one Voyager database, but
we also OAI harvest other collections.
An easy trick is to just prepend a unique prefix before the
normal bib-id.
For example - we have 'SLEDGE143' and '143' records:
   http://catalog.lib.auburn.edu/vufind/Record/SLEDGE143 
   http://catalog.lib.auburn.edu/vufind/Record/143 

I think you'll have to hack at least 2 things to get this scheme to
work:

  *. Modify Drivers/Voyager.php to manage connections
    to multiple Oracle servers, and choose a connection
    based on the id-prefix.

  *. Modify the solrmarc vufind.properties file, and replace
          id = 001, first
    with something like
          id = my001WithPrefixFunction
    ... something like that.

I'd like to do something like this myself, but it keeps falling off
the end of my TODO list.
Let me know if you need help implementing this, and I'll set aside a 
day or two to give it a try.

Cheers,
Reuben
    
 

2010/10/18-21 - fish data

It was a quiet week for me at the library. I spent almost all my time working on the multi-step zip-submission forms for the minnows morphology project. The code is coming along well - it's just taking some time. I'll keep at it next week.

2010/10/04-07,11-14 - lazy blogger

I haven't kept up with this bLog the last couple weeks, but there's not much to tell anyway. I've mostly just been working on my usual assortment of projects.

• I finished a v2v refactor moving the CLI interface over to littleware's lgo infrastructure, and adding Clint's -halt and -continue flags. Clint kicked off an import last week, and it looks like it runs like a charm.
• I sent Jon Armbruster a link to a prototype data-submission form for the cyprinella morphology repository. After a little back and forth we added a few more requirements for the submission to support. I'll work on that next week.
• After Claudine finishes her pass over the ACES data I think we'll be set to import that collection into the repository.
• We met with Troy from fisheries who has a document collection he'd like us to index into vufind. It looks like that will be easy for us to do - Troy will send us an XML data file, and we'll run that through our XSLT pipeline. There was a little bit of "should we do this" goofiness that the librarians had to talk themselves through, but I was able to avoid most of that.
• Along the same lines - there was some nervous collapse in the vufind committee, who decided it's outside their "charter" to test vufind integration with article level search. The librarians will probably form yet another committee to consider article level search.
• I'm going to check with Tony next week about setting up podcast feeds for some of the video he's posting to vimeo. That should be easy to do, but Tony might prefer to only stream the content rather than make files available for download. We'll see.
• I also need to touch base with Kathie Mattox at the honors college next week. It's getting close to the end of the semester - does she still plan to allow honors students to deposit into the online thesis repository ?
• We reviewed the 20 submissions for the library programmer position, and narrowed the list down to 10 phone interviews. They're all good candidates, but I'm not excited about hiring more staff at the library.

2010/09/27-30 - moving through jello

I spent most of this week working on a submission tool for the minnows morphology project. The following e-mail describes what I'm implementing.

... Reuben Pasquini 09/30/10 1:01 PM ...
Hi Jon,

I've been looking at how to manage submissions to the morphology repository,
and I have some ideas to bounce off you.
I'd like to implement the following process.

  *. We'll setup a custom submission form that only allows a user
    to upload a single .zip file.
    The .zip file may contain an arbitrary number of .tps files,
    and the .jpg images that go with them.

  *. A user uploads the zip file, and fills out a form of metadata.

  *. The server automatically unzips the zip file,
    and breaks it down into individual items like
       http://131.204.172.126:8080/minnows/handle/123456789/10 
   , and submits the items for review after the user
   verifies that everything looks ok.

  *. The reviewer approves or rejects each individual item.     

That's the direction I'm working in now - sound ok ?

Cheers,
Reuben

BTW - If you want to allow excel spread-sheeting in addition to .tps files, 
    then send me instructions on how the data in a spread sheet
    maps to the data in a .tps file, and I'll code that in too.

I felt like I was moving through jello this week - writing code and doing little things, but nothing released to users. I did contribute (see below) to an e-mail exchange about testing article-level search in our vufind catalog.

... Reuben Pasquini 09/30/10 10:36 AM ...
Hi Nancy!

Thanks for taking the time to write down your thoughts - you have
great ideas.  I'll help stir the pot a little - see if we can get her to boil!

*. Your argument about wanting students
           "
             to think more critically about
             where and what they're searching for information
            "
      made me think that I actually want the opposite.
      I don't want to think at all - just give me what I want.

*. Your point about the apparent overlap between the vufind, voyager-2, 
    and EBSCO efforts is well taken.
    It makes sense to have vufind, voyager-2, and WAG
    under one umbrella.
    I don't think it's a big problem though.
    Clint or Tony are in every meeting, so they keep things connected.
    I think they like meetings.

* Google Scholar:
      http://scholar.google.com/intl/en/scholar/about.html 
   "
Google Scholar aims to rank documents the way researchers do, weighing the full text of each document, where it was published, who it was written by, as well as how often and how recently it has been cited in other scholarly literature.
   "
    Sounds awesome.

*. "Are we using our technical and human resources correctly?"
    Are you having doubts only now ?
    You probably still leave cookies out for Santa ...


Cheers,
Reuben

... "Nancy Noe"  9/29/2010 8:56 PM ...
Hi Marliese

Working tonight at the Village.  I've been able to play around with the
Villanova site a bit, but haven't given it the time or consideration it
deserves.  These are my initial impressions.  In the spirit of
full-disclosure, I should probably say that my comments come from a very
specific information literacy philosophy.

One of the concerns I have with this kind of searching is the same issue
I have with federated searching.  While I understand the 'google-like'
feel of such a search, I want students to think more critically about
where and what they're searching for information.  There was a recent
discussion on the  instruction list serv - are people teaching federated
searching/discovery services? While a couple of people said 'yes', the
majority replied in the negative.   Again, it comes back to helping
students to learn to move away from the 'everything possible' approach,
to one that asks them to consider what disciple/subject specific
resources might provide them with higher quality academic/scholarly
material.  I want them to think critically about their information need
before they start typing in poorly constructed, broad-based keyword
searches, especially when looking for articles.   For instance, I taught
an honors ENGL1127 today, working on literacy issues.  Within the EBSCO
suite, I was able to have them select ASP and three education related
databases.  That's the kind of thing I'm going to teach in class. 
That's where I'm going to fall on the issue.

Technically, I found that in the searches I conducted, there were a lot
of 'clicks' one had to go through before you got to the actual article. 
In two instances, there was a link to full-text that failed (and that
was prior to asking for log-in information.)

Vendors are creating systems that in a way compete with Google Scholar,
yet I suppose there's a 'cost' for these systems.  Perhaps I don't fully
understand, but won't they be charging us for the privilege of using
metadata when we already pay for access to the info through our database
subscriptions?  Does this give us permanent access to the articles? 
Faculty tell me that they now use Google Scholar as the starting point
for their research.   Many schools have a link to Google Sciholar off
their library homepage.  Would this serve us just as well?   I may not
be as informed as I need to be on cost and ownership issues.

Should the VuFind cmte investigate?  Maybe.  We're still working on
trying to make the catalog work (advanced searching?) and trying to
determine if we should recommend as primary.  To be honest, the past
couple of weeks I've found myself reverting back to the 'classic'
catalog to answer journal title questions.  I haven't settled on
questions I have about VuFind as a catalog.   I'm not saying that we
shouldn't be exploring new systems/new technologies.  We should.  Is
this the right cmte to take this forward?  We have the new voyager cmte,
and the EDS cmte, and the vufind cmte?  Are we using our technical and
human resources correctly?  Is there a way to better optimize time and
talents, especially since it seems that all of these are converging.

So, those are my two cents this evening.  Again, I'll miss the meeting
tomorrow.

Thanks

Nancy

... "Marliese Thomas"  09/29/10 2:52 PM ...
Hi everyone, 

Here's some background about the opportunity I mentioned at our last
meeting.

We've been talking for a while about taking VuFind to the next level by
including article-level metadata in it.  Indeed, at our August 9th
meeting, Bonnie volunteered to contact vendors about getting a batch of
article-level metadata that we could run through Vufind in order to see
how this might work.

There have been a couple of new developments since that meeting. 
First, Bonnie, Marcia, Aaron, and I met with Jane Burke and Mary Miller
of Serials Solutions a couple of weeks ago.  In the course of that
meeting, Jane told us that Serials Solutions has developed a programming
interface that allows other discovery services, including VuFind, to
search Summon and display results from it.  This functionality allows
VuFind to do article-level searches in Summon without the host library
having to ingest, store, and index large article-level metadata sets
provided by vendors.

To see how the VuFind-Summon combination works in practice, go to
Villanova's VuFind catalog at:

https://library.villanova.edu/Find/Search/Home 

Do a search.  In addition to the familiar-looking VuFind results,
you'll see a box in the upper right-hand corner called "Top results from
Articles & more", with links.  "Articles & more" = Summon.  One example:
https://library.villanova.edu/Find/Search/Results?lookfor=cat&type=&search=catalog+AllFields&submit=Find 


The day after we met with the Summon folks, EBSCO announced that EBSCO
Discovery Service has a similar functionality.  Since we're a full-level
development partner for EDS, we can test this feature with VuFind at no
cost to us.  (Of course, we'd have to buy EDS, or Summon, or a similar
product, in order to offer article-level content to our users as part of
our regular services.) Since then, I have also heard that Ex Libris is
offering this same functionality for their discovery product, Primo
Central. Obviously, this is becoming a common and viable way for
libraries to leverage use of their software contracts.

I believe it would be worth exploring how VuFind works with a
full-featured commercial discovery service like EDS.  In fact, Bonnie
gave Aaron and me the go-ahead to take a shot at getting a VuFind-EDS
test installation working in time for my presentation this weekend at
LITA.  I'd be interested in hearing your thoughts on these developments
at our meeting tomorrow, or when you have had a chance to explore the
Villanova catalog.

Thanks,

Marliese



Marliese Thomas
Database Enhancement Librarian
RBD Library
Auburn University
 
231 Mell Street
Auburn, AL 36849
 
334.844.8171
mst0001@auburn.edu 

Finally, Thursday was Penny's last day. Enjoy your retirement, Penny!