Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
maxLevel5


Info
titlePicking up where you left off

If you have already completed the installation steps below and are just trying to run the importer, follow the steps below. Otherwise skip ahead to "What You'll Need" and proceed from there.

  1. Change to the directory where you created your virutalenv. If you're having a hard time remembering where this was, use the command:

    Code Block
    find . -type d -name venv

    to locate the virtualenv directory. You can copy/paste that path and use the command:

    Code Block
    cd PATH/TO/VENV
    cd ..

    to change to the right directory.

  2. Activate the virtualenv using the following command:

    Code Block
    source venv/bin/activate

    If this worked your prompt should be prefixed with "(venv)", like:

    Code Block
    (venv) sam@multivac:~$


  3. Get a up to date cookie from EditThisCookie (Step 2 below).

  4. Change into the mandala-bii directory:

    Code Block
    cd mandala-bii


  5. You should now be ready to run the importer!


What You'll Need

First: 

  • brew
  • python
  • pip

Then:

  • virtualenv
  • virtualenwrapper
  • imagemagick
  • dcraw
  • exiftool

Also:

...

Info
iconfalse
titleSave for later:

The password you created.

2. Use EditThisCookie  to get the key you need to access Mandala Images

Final result: A KEY and VALUE

...

Info
iconfalse
titleSave for later:

The KEY and VALUE you got in step 5.


3. Set up your package installers and python

Final result: python, pip, and brew installed on your computer

...

You should see the version of pip that's been installed. 

4. Install other stuff you need

Final result: These packages installed onto your computer

...

Code Block
pip install pyexifinfo

5. Create a directory for the importation code 

Using the Terminal, you can create and move into 'directories'. This is just creating and moving into folders on your computer using the command line. For simplicity's sake, we're going to create a folder directly in your documents called bhutan-import

...

If you open up Finder, you'll see there's a folder called 'bhutan-import' in your Documents.

6. Get the importation code and XML files

Run this code to clone the importation code into bhutan-import.

...

A folder called mandala-bii will show up in bhutan-import once the code is cloned. 

For Images Cataloged with Media Pro

If your images have embedded metadata generated with Adobe Bridge, skip to the next step (7). If the images were managed using Mediap Pro, contact Veronica KuhnContact Sam Chrisinger, who will link you to an .xml file from Box. Download the file.

Put the downloaded file inside the mandala-bii folder that you created in the previous step. You can use the good ol' click and drag method, no need to use the Terminal! 

...

Info
iconfalse
titleSave for later:

The name of the xml file you downloaded.

7. Set up the virtual environments

Back in the Terminal, run: 

...

If you've done this correctly, you should see (VENV NAME) at the beginning of the bash prompt. 

8. Set up the importation code

Enter the mandala-bii directory by running the following: 

...

Code Block
pip install -r requirements.txt

9. Create a collection

In Mandala, create a collection or subcollection to house the images you're importing. As you do this, check the URL in the collections editor. This lets you find the collection id, which you'll need to direct the import to that collection. For example, for the Chuma Laptse collection, I can go into the editor and see the URL is https://images.shanti.virginia.edu/node/1419391/edit, so the collection id is 1419391. 

Info
iconfalse
titleSave for later:

The collection id.

10. Run the importer

For Images Cataloged Using Adobe Bridge 

Use the following code, with some substitutions: 

  • KEY and VALUE are codes you obtained in step 2. 
  • YOUR_BOX_USER is your box username, which should be your UVa email address
  • YOUR_BOX_PASS is the password you created in step 1. 
  • /IMAGE/FILE/PATH needs to be replaced with the path to the files in UVaBox. 
  • COLLECTION_ID is the id of the collection where you're importing the images. 
Code Block
languagetext
python import.py -s Encoded -c KEY=VALUE -i "/IMAGE/FILE/PATH" -u https://images.shanti.virginia.edu/admin/content/bulk_image_import/api -v -cid COLLECTION_ID -cv -cw ImageMagick --ftp --ftp_user YOUR_BOX_USER --ftp_pass YOUR_BOX_PASS

For Images Cataloged Using MediaPro

Use the following code, with some substitutions: 

  • XML_FILE_NAME is the name of the file you downloaded in step 6. Please make sure you've moved this file to the mandala-bii folder, or the importation will fail!
  • KEY and VALUE are codes you obtained in step 2. 
  • YOUR_BOX_USER is your box username, which should be your UVa email address
  • YOUR_BOX_PASS is the password you created in step 1. 
  • Bhutan Year X needs /IMAGE/FILE/PATH needs to be replaced with the year path to the files in UVaBox. 
  • COLLECTION_ID is the id of the collection where you're importing (ex: Bhutan Year 4). This should match the xml file. The folder structure needs to be identical to what's on Box, so refer back to the original folder to check this.  the images. 

Code Block
languagetext
python import.py -s MediaPro -x "XML_FILE_NAME.xml" -c KEY=VALUE -i "/Image UploadsIMAGE/Bhutan/Bhutan Year X/XML_FILE_NAMEFILE/PATH" -u https://images.shanti.virginia.edu/admin/content/bulk_image_import/api -v -cid 619101COLLECTION_ID -cv -cw ImageMagick --ftp --ftp_user YOUR_BOX_USER --ftp_pass YOUR_BOX_PASS

...

You can ignore any other messages as long as the importation is still running. Any imported images should be immediately visible at at your collection on Mandala.

Troubleshooting

Fixing broken images (that have already been imported)

Sometimes an image is imported incorrectly during importation. This can occur because of an issue with converting the image, or for other reasons. The importer script provides a way to selectively re-upload or "repair" an already imported image. Use the steps below to do so:

Get a list of the images you want to repair- you will need a CSV file with an entry containing the node id, task id, and filename for all of the images. The easiest way to get this list for a catalog is to:

Log in to Acquia Cloud and visit the detail page for the Prod environment (https://cloud.acquia.com/app/develop/applications/14584a7f-64a0-4816-ad7f-b1d9a6da429a/environments/23134-14584a7f-64a0-4816-ad7f-b1d9a6da429a)

Copy the "SSH URL" to your clipboard, and from a terminal run:

Code Block
ssh SSH URL

replacing SSH URL with your clipboard contents.

Run: 

Code Block
cd /var/www/html/shanti.prod/docroot/sites/images/


You are about to run some commands in MySQL. Please proceed with caution. Updates and deletes are committed to the Production database immediately.

To find your catalog name run: 

Code Block
drush sqlq "select distinct(catalog) from bulk_image_import_task;"


Then run: 

Code Block
drush sqlq "select node_id, tid, filename from bulk_image_import_task where catalog = 'YOUR CATALOG NAME';" > /home/shanti/import.tsv


Enter CTRL-d to exit the SSH session

To copy the file down locally, run:

Code Block
scp shanti.prod@ded-18858.prod.hosting.acquia.com:/home/shanti/import.tsv ~/Downloads/import.tsv


Then run:

Code Block
cat ~/Downloads/import.tsv | tr "\\t" "," > ~/Downloads/import.csv


You now have a file (import.csv) that contains a list of node id, task ids, and filenames for each entry in your catalog. You can now run the importer's repair script:

Code Block
python import.py -s Repair --csv "import.csv" -c COOKIE=VAL -i "PATH TO IMAGES FOLDER" -u https://images.shanti.virginia.edu

...

/admin/content/bulk_image_import/api -v --ftp --ftp_user YOUR USERNAME --ftp_pass 'XXXXXX' -cv -cw ImageMagick