Tar Utilities Python Module

Utilities and classes for unbundling and archiving a tar file.

class pacifica.ingest.tarutils.FileIngester(hashtype, hashcode, file_id)

Class to ingest a single file from a tar file into the file archives.

__init__(hashtype, hashcode, file_id)

Constructor for FileIngester class.

file_id = 0

fileobj = None

hashval = None

read(size)

Read wrapper for requests that calculates the hashcode inline.

recorded_hash = ''

server = ''

upload_file_in_file(info, tar)

Upload a file from inside a tar file.

validate_hash()

Validate that the calculated hash matches the hash uploaded in the tar file.

exception pacifica.ingest.tarutils.HashValidationException

Class to capture hashsum validation failures.

class pacifica.ingest.tarutils.MetaParser

Class used to hold and search metadata.

__init__()

Constructor.

clean_metadata()

clean /data from filepaths.

file_count = -999

file_obj_count(meta_list)

Count the file objects in metadata and keep the count.

files = {}

get_fname(file_id)

Get the file name from the file ID.

get_hash(file_id)

Return the hash string for a file name.

get_subdir(file_id)

Get the sub directory element from the file ID.

load_meta(tar, job_id)

Load the metadata from a tar file into searchable structures.

meta = None

meta_str = ''

post_metadata()

Upload metadata to server.

read_meta(metafile, job_id)

Read the metadata from metafile and assume it’s good.

start_id = -999

transaction_id = -999

class pacifica.ingest.tarutils.TarIngester(tar, meta)

Class to read a tar file and upload it to the metadata and file archives.

__init__(tar, meta)

Constructor for TarIngester class.

ingest()

Ingest a tar file into the file archive.

meta = None

tar = None

pacifica.ingest.tarutils.file_count(tar)

Retrieve the file count for a tar file.

Does not count metadata.txt as that is not uploaded to the file archive

pacifica.ingest.tarutils.get_clipped(fname)

Return a file path with the data separator removed.

pacifica.ingest.tarutils.open_tar(fpath)

Seek to the location of fpath, returns a file stream pointer and file size.

pacifica.ingest.tarutils.patch_files(meta_obj)

Patch the files in the archive interface.