Off-Site Backups with Bacula

 

Bacula, the Open Source Network Backup Solution

Bacula is a great Backup Solution for any infrastructure out there, if you don’t know this tool, I really recommend reading about it. I’ve been using it for over 2 years now and when I thought I had learned everything there was to learn about it, I was asked to do one more thing with our backups: store them on a off-site location for security. After all, what’s the point in having data backups stored in your company if a flood or a fire will destroy everything, including your backups?

“What’s the problem?”, you may ask… “Just take your daily backups tapes to an off-site location!”, you might say… If only life was that simple…

I needed those daily backups tapes on-site, since our helpdesk team was constantly asking us to restore some files from the Samba Share server, because the user “swears the file just disappeared from the server”… Yeah, right…

So, how could I possibly have a secure off-site backup of my data and, at the same time, have them available to me to make daily restores in my company?

 

Bacula and Copy Jobs

Copy Jobs to the rescue! Man, what a great feature. If you’re familiar with Bacula (which I hope you are, since this posts expects you to), you know that every single backup that Bacula does is actually a Job to it.

Basically, a Copy Job is a special Job (like an Admin Job) that takes a list of JobId based on a criteria chosen by you (will see them further on) and makes a copy of those Jobs from their original Pool to a different Pool.

I think you’ll understand better with examples…

 

The Scenario

Currently, I have the basic three Pools for my Backups: one for my Full Backups (pool.full), one for my Differential Backups (pool.diff) and the last one for my Incremental Backups (pool.inc). Their configurations is as the following:

Pool {
    Name = pool.full
    Pool Type = Backup
    Storage = st.tpc
    Volume Use Duration = 6 months
    Volume Retention = 1 year
    Scratch Pool = scratch.tpc
    RecyclePool = scratch.tpc
    File Retention =  1 year
    Job Retention =  1 year
    Cleaning Prefix = "CLN"
}

Pool {
    Name = pool.diff
    Pool Type = Backup
    Storage = st.tpc
    Volume Use Duration = 1 week
    Volume Retention = 6 months
    Scratch Pool = scratch.tpc
    Recycle Pool = scratch.tpc
    File Retention =  1 year
    Job Retention =  1 year
    Cleaning Prefix = "CLN"
}

Pool {
    Name = pool.inc
    Pool Type = Backup
    Storage = st.tpa
    Volume Use Duration = 1 month
    Volume Retention = 3 months
    Scratch Pool = scratch.tpa
    RecyclePool = scratch.tpa
    File Retention =  1 year
    Job Retention =  1 year
    Cleaning Prefix = "CLN"
}

The details about my Backup Policy is not important, what you have to know is: I have 3 tape libraries named st.tpa, st.tpb and st.tpc. The first I’m using for my Incremental Backups and the last I’m using for my Differential and Full Backups. The second, st.tpb, I going to use for my Copy Jobs.

This is the first and possibly the most important note there is to know before starting using Copy Jobs: YOU MUST HAVE TWO DIFFERENT STORAGES IN ORDER TO USE COPY JOBS. For example: if you have a File Storage configured in your Bacula Storage Daemon where you send all your backups to, you can copy those jobs to an second File Storage or even to a Tape Storage, but you can’t copy your jobs to same storage location where you send your original jobs.

In my case, I had the Tape Storage st.tpb lying around with no use, so far.

I also had very simple Jobs and Schedules, which will execute a Full Backup twice a year, a Differential backup weekly and a Incremental Backup on a daily basis:

Schedule {
  Name = sch.default
  Run = Level=Full Pool=pool.full jan on 1st sat at 21:00
  Run = Level=Full Pool=pool.full jul on 1st sat at 21:00
  Run = Level=Differential Pool=pool.diff FullPool=pool.full on 1st sat at 21:00
  Run = Level=Incremental Pool=pool.inc FullPool=pool.full sun-fri at 21:00
}

Client {
  Name = client.test
  Address = test.example.dom
  Catalog = cat.default
  Password = "SeCrEt"
  File Retention = 1 year
  Job Retention = 1 year
}

Jobs {
  Name = job.test
  Type = Backup
  Client = client.test
  Write Bootstrap = "/var/spool/bacula/jobs/bootstrap.%c.%n.bsr"
  FileSet = fs.default
  Messages = msg.default
  Pool = pool.full
  Schedule = sch.default
  Max Start Delay = 72h
  Spool Data = yes
  Allow Duplicate Jobs = no
  Cancel Lower Level Duplicates = yes
}

You can see that the Schedule is what defines which pool to use for each level of backup (Inc, Diff and Full). So, here what we’ve got so far:

1. Full Backups go to Pool pool.full, residing on Storage st.tpc (Tape Library);
2. Diff Backups go to Pool pool.diff, also residing on Storage st.tpc (Tape Library);
3. Inc Backups go to Pool pool.inc, residing on Storage st.tpa (Tape Library);

 

The Backup of the Backups

Let’s start making Copy Jobs of my Full Backups, you’ll see that applying to the rest of the backups is very trivial. First, we need to create a Pool to throw the Copy Jobs, and here it is:

Pool {
    Name = pool.offsite
    Pool Type = Backup
    Storage = st.tpb
    Volume Use Duration = 1 day
    Volume Retention = 6 months
    Scratch Pool = scratch.tpb
    RecyclePool = scratch.tpb
    File Retention =  1 year
    Job Retention =  1 year
    Cleaning Prefix = "CLN"
}

It is a regular Pool just like any other, with just a few notes: first, I’m using the “st.tpb” Storage for this Pool, remember that I have to use a different Storage for the Copy Jobs.

Second, the “Volume Use Duration” is 1 day because I’ll run the Copy Job once a week and, every week, it will start to use a different tape, because the one from the week before will be mark as “Used”. That’s specifically important to me because I’ll remove the tapes from this Storage to take them to an off-site location and, without this option, I would end up with “mount volume” error messages because Bacula would try to use the Tape that removed the week before on this week’s Copy Job.

How do you tell Bacula to Copy it’s Full Backups to this Pool? First, you must tell the Full Pool where to put the Copy Jobs, by adding the line “Next Pool” to it:

Pool {
    Name = pool.full
    Pool Type = Backup
    Storage = st.tpc
    Volume Use Duration = 6 months
    Volume Retention = 1 year
    Scratch Pool = scratch.tpc
    RecyclePool = scratch.tpc
    File Retention =  1 year
    Job Retention =  1 year
    Cleaning Prefix = "CLN"
    Next Pool = pool.offsite
}

Second, you must create a special Copy Job:

Job {
    Name = job.copyjob.full
    Type = Copy
    Pool = pool.full
    Selection Type = PoolUncopiedJobs

    Messages = msg.default
    Schedule = sch.none
    Client = client.none    
    FileSet = fs.none    
}

For a Copy Job, the parameters “Client” and “FileSet” really don’t matter at all. They are just there because the “Job” entry requires them. That’s it.

What matter is:

1. Type: use the special keyword “Copy”;
2. Pool: which Pool the Jobs will be copied FROM (the Pool it self tells where they will be copied TO, with the “Next Pool” directive);
3. Selection Type: the option “PoolUncopiedJobs” basically tells Bacula to Copy all Jobs on this Pool that yet has not been copied before;

There are other ways of telling Bacula which Jobs it should make a Copy, but for me, making a Copy of every single Job of the Pool was just perfect. But if you like, you can even make a SQL Query to select which Jobs Bacula should make a Copy of.

 

Running the Bastard!

After configuring all that, since I’m using a fake Schedule directive for my Copy Job, I have to run it manually. I recommend doing this for the first few times just to see if everything goes fine. Dan Langille said this on his blog: walk first, run later.

So, to finally run your Copy Job, just execute on “bconsole”:

*run job=job.copyjob.full
Run Copy job
JobName:       job.copyjob.full
Bootstrap:     *None*
Client:        client.none
FileSet:       fs.none
Pool:          pool.full (From Job resource)
Read Storage:  st.tpc (From Pool resource)
Write Storage: st.tpb (From Storage from Pool's NextPool resource)
JobId:         *None*
When:          
Catalog:       cat.default
Priority:      10
OK to run? (yes/mod/no):

On this output, notice the “JobId”? It’s blank because it will select the JobId’s to make the Copy of at run time. Fantastic, huh? Since I’m using the directive “PoolUncopiedJobs”, this single Job will create a Copy Job for every Full Backup Job that is stored on the the Pool “pool.full”.

Another thing to notice: “Read Storage” and “Write Storage”. Once again, THEY MUST BE DIFFERENT!

Once you strike “yes”, the madness begins! You’ll see something similar to this on your log:

JobId 30: The following 2 JobIds were chosen to be copied: 12,13
JobId 30: Job queued. JobId=31
JobId 30: Copying JobId 31 started.
JobId 30: Job queued. JobId=33
JobId 30: Copying JobId 33 started.

By the time I ran this Copy Job, I had two Full Backup on my Pool “pool.full” and, guess what? Their JobId’s were consecutively 12 and 13! Like I said before, Bacula creates a different Copy Job for every Job that needs to be copied. Since I had two Full Backups, it creates two Copy Jobs with JobId 31 and 33.

And that’s it! Now just wait for your Copy Jobs to finish and keep checking your logs for any errors. In my case, I remove the Tapes used for Copy Jobs from the st.tpb Tape Library and take them to the off-site location, and replace them with new ones. Just remember that, every time you replace your Tapes, you must label them with “label barcodes” (or simply “label”, if your Tape Library doesn’t support barcodes) and always, ALWAYS, do an “update slots”, to tell Bacula that those tapes have been replaced on your Tape Library.

What about the Differentials and Incremental backups? They will work exactly the same way as the Full Backups, just add an “Next Pool” directive on their Pool. You can even use the same Pool you’re using for copying the Full Backups, like this:

Pool {
    Name = pool.diff
    Pool Type = Backup
    Storage = st.tpc
    Volume Use Duration = 1 week
    Volume Retention = 6 months
    Scratch Pool = scratch.tpc
    Recycle Pool = scratch.tpc
    File Retention =  1 year
    Job Retention =  1 year
    Cleaning Prefix = "CLN"
    Next Pool = pool.offsite
}

Pool {
    Name = pool.inc
    Pool Type = Backup
    Storage = st.tpa
    Volume Use Duration = 1 month
    Volume Retention = 3 months
    Scratch Pool = scratch.tpa
    RecyclePool = scratch.tpa
    File Retention =  1 year
    Job Retention =  1 year
    Cleaning Prefix = "CLN"
    Next Pool = pool.offsite
}

Meaning that even your Differential and Incremental Backups stored on the pool.diff and the pool.inc would be copied to the pool.offsite Pool. You just need to create a Copy Job for each one:

Job {
    Name = job.copyjob.inc
    Type = Copy
    Pool = pool.inc
    Selection Type = PoolUncopiedJobs

    Messages = msg.default
    Schedule = sch.none
    Client = client.none    
    FileSet = fs.none    
}

Job {
    Name = job.copyjob.diff
    Type = Copy
    Pool = pool.diff
    Selection Type = PoolUncopiedJobs

    Messages = msg.default
    Schedule = sch.none
    Client = client.none    
    FileSet = fs.none    
}

And run them manually like we did with the “job.copyjob.full” before. Off course, you can create a “real” Schedule resource to automatically run you Copy Jobs daily, for example. I’m just being extra careful, testing if everything is going alright before letting them run in the wild.

 

Now What?

Now, in case of a disaster and you loose your original Backup Tapes, how do you recover your data from the Copy Jobs Tapes? I’m still figuring it out, but very soon I’ll make a new post talking about it.

That’s it for now! Good luck!

EDIT: I’ve created a post explaining how to restore data from the Copy Jobs, check it out here.