The Library is closed onsite, open online. See updates here.

Agile thinking - how can we create a bulk-ingest tool?

The Library has been engaged in the digitisation of the physical collection for many years and has an active born digital acquisition programme.  What does this mean?  Thousands of files that we need to ingest and preserve in our preservation system and the numbers just keep going up.

Recently the Library began development on a software tool that will manage the bulk ingest of digital files into our digital preservation system, Rosetta.  The tool, PanDA (Preservation and Digital Access), will include a web based user interface, more efficient handling of errors, detailed reporting and efficient processing of high volumes of files by utilising multiple processor cores concurrently.

For the development of PanDA, the technical team decided to use an Agile approach collaborating with subject matter experts and Library staff. This was a new approach for the Library, having favoured more traditional project management methods in the past.

PanDA, SLNSW bulk ingest tool user interface

User interface for PanDA, SLNSW bulk ingest tool 

What is Agile project methodology?

It is a project management process or framework that allows for greater flexibility than the more traditional waterfall methods. The process allows for high levels of collaboration between the business area or subject matter experts and the development team. The process is iterative and incremental with a focus on giving the SME staff an opportunity to test and approve the product at every stage of the development. If there are problems or challenges, then changes can be made quickly to fix the issue.


I have enjoyed the Agile process and can see how the approach may benefit other projects in the Library.  Being part of the planning, the ongoing testing process and regular meetings we are more accountable as business owners of a system. Rather than being delivered a large product at the end of a traditional process, we could adapt, adjust our expectations and be flexible to get the best product. 

Damien Cassidy, Digital Media Technical Analyst and subject matter expert, has enjoyed the Agile approach for "… the relevance, fluidity, flexibility and inclusiveness. It's less constricting than the usual framework which often feels like a competing priority and disconnected from the project outcomes".

So how do developers feel about using the Agile approach at the Library?

Peter Brotherton, Systems and Applications Developer and the SCRUM master (no Project Managers here),  feels that the Agile approach  "... has empowered our development team to quickly iterate in response to feedback from business owners. It allows us to focus our efforts on building software which meets evolving functional requirements. In traditional waterfall projects, there is limited opportunity for business owner engagement, with agile software development the business owners are involved every step of the way. As a result, the end software fulfils practical business needs and reliability is higher due to automated testing and a focus on User Acceptance Testing."

We are six months int (that is 8 SPRINTS for the Agile experts) and we are all very happy with the project approach and of course the PanDA tool.  When we go live we will be having a Panda party to celebrate, not only a tool that is going improve our through put and workflows, but all the hard work that we have all put in over the past six months.

Panda Biscuit by Andrew J

Panda Biscuit  Photograph by Andrew J  CC BY-NC-SA 2.0


The Library aims to release PanDA in the coming months and will make it available to anyone who is interested in bulk ingest of digital collections. 


- Written by Joanna Fleming, Digital Curation Specialist