OpenSCAP XSLT performance improvements for faster SSG builds

As I contribute more and more patches to SCAP Security Guide I got increasingly frustrated with the build speeds. A full SSG build with make -j 4 took 2m21.061s and that’s without any XML validation taking place. I explored a couple of options how I could cut this time significantly. I started by profiling the Makefile and found that a massive amount of time is spent on 2 things.

Generating HTML guides


We generate a lot of HTML guides as part of SSG builds and we do that over and over for each profile of each product. That’s a lot of HTML guides in total. Generating one HTML guide (namely the RHEL7 PCI-DSS profile from the datastream) took over 3 seconds on my machine. While not a huge number this adds up to a long time with all the guides we are generating. Optimizing HTML guides the first thing I focused on.

I found that we are often selecting huge nodesets over and over instead of reusing them. Fixing this brought the times down roughly 30%. I found a couple other inefficiencies and was able to save an additional 5-10% there. Overall I have optimized it roughly 35-40% in common cases.

During the optimization I have accidentally fixed a pretty jarring bug regarding refine-value and value selectors. We used to select a big nodeset of all cdf:Value elements in the entire document, then select all their cdf:values inside and choose the last based on the selector. This is clearly wrong because we need to select the right cdf:Value with the right ID and then look at only its selectors. Fixing that make the transformation faster as well because the right cdf:Value was already pre-selected.

Old XSLTs:

$ time ../../../shared/utils/ -j 1 --input ssg-rhel7-ds.xml
real 0m16.736s
user 0m16.349s
sys  0m0.397s

New XSLTs:

$ time ../../../shared/utils/ -j 1 --input ssg-rhel7-ds.xml
real 0m11.203s
user 0m10.836s
sys  0m0.379s

EDIT: I found more optimization opportunities, latest data as of 2016-08-10:

real 0m3.399s
user 0m2.986s
sys  0m0.409s

I won’t be redoing the entire test-suite and all the graphs but the final savings are much better than it shows in the graph. Generating all RHEL7 SDS guides takes less than 2 seconds on my machine after the optimizations.

Transforming XCCDF 1.1 to 1.2


It took 30 seconds on my machine to transform RHEL6 XCCDF 1.1 to 1.2. That is just way too much for a simple operation like that. Clearly something was wrong with the XSLT transformation. As soon as I profiled the XSLT using xsltproc --profile I found that we select the entire DOM over and over for every @idref in the tree. That is just silly. I fixed that by using xsl:key and using the very same @idref to element mapping for all lookups. This saved a lot of time.

Doing the RHEL6 XCCDF 1.1 to 1.2 transformation with old XSLTs

real 0m34.635s
user 0m34.585s
sys  0m0.047s

Doing the RHEL6 XCCDF 1.1 to 1.2 transformation with new XSLTs

real 0m0.619s
user 0m0.573s
sys  0m0.045s

The numbers were similar for the RHEL7 XCCDF 1.1 to 1.2 transformation.

Final results for the SSG build

I started with 2m21.061s and my goal was to bring that down to 50%. The final time on my machine after the optimizations with make -j 4 is 1m4.217s. Savings of roughly 55%. Most of those savings are in the XCCDF 1.1 to 1.2 transformation that we do for every product.

The savings are great on my beefy work laptop (i7-5600U) but we should benefit even more from them on our Jenkins slaves that aren’t as powerful. I have yet to test how much they would help there but I estimate it will be 10 minutes for each build.


When I suggested to deploy these improvements on our Jenkins slaves, Jan Lieskovsky brought up an important point about correctness. We decided to diff old and new guides and old and new XCCDF 1.2s to be sure we aren’t changing behavior. Please see the attached ZIP file for a test case I created to verify that we haven’t changed behavior. During the process of creating this test case I discovered that I have accidentally fixed a bug mentioned above 🙂 To silence the diffs I have introduced just this bug into the new XSLTs I used. This made the performance slightly worse so keep that in mind when looking at the numbers.

Doing the RHEL6 XCCDF 1.1 to 1.2 transformation with old XSLTs

real 0m34.635s
user 0m34.585s
sys  0m0.047s

Doing the RHEL6 XCCDF 1.1 to 1.2 transformation with new XSLTs

real 0m0.619s
user 0m0.573s
sys  0m0.045s

Diffing old_xslt_output/ssg-rhel6-xccdf-1.2.xml and new_xslt_output/ssg-rhel6-xccdf-1.2.xml
The files are the same.

Doing the RHEL7 XCCDF 1.1 to 1.2 transformation with old XSLTs

real 0m33.146s
user 0m33.089s
sys  0m0.050s

Doing the RHEL7 XCCDF 1.1 to 1.2 transformation with new XSLTs

real 0m0.749s
user 0m0.702s
sys  0m0.047s

Diffing old_xslt_output/ssg-rhel7-xccdf-1.2.xml and new_xslt_output/ssg-rhel7-xccdf-1.2.xml
The files are the same.
Doing the RHEL6 and 7 SDS HTML guide transformations with old XSLTs

real 0m39.104s
user 0m38.605s
sys  0m0.491s

Doing the RHEL6 and 7 SDS HTML guide transformations with new XSLTs

real 0m28.974s
user 0m28.531s
sys  0m0.433s

Diffing old_xslt_output/guides_for_diff and new_xslt_output/guides_for_diff
No differences.

UPDATE: Jenkins build times (2016-08-12)

Here is a graph of Jenkins build times, you can see how the build times gradually went lower as optimizations got onto the Jenkins slaves. There are occasional build time spikes caused by load when multiple pull requests were submitted at once but overall the performance has been improved.

Combine SCAP tailoring file and datastream into a single file

Many users customize their SCAP content before use. Usually they use SCAP Workbench. When they are done they end up with the original source datastream and a customization file. If they are scanning using the oscap tool or SCAP Workbench they can use them as they are. If they are however using Red Hat Satellite 6 to do their SCAP scans they cannot upload the 2 files to form a single policy. Instead they need to somehow combine the tailoring and datastream to get a single file. In this blog post we will explore how to do just that.

Option 1: Manual surgery (not recommended)

The first option is to take the Profile from the tailoring file and insert it into the XCCDF Benchmark. Let us see how the tailoring file looks like:

<?xml version="1.0" encoding="UTF-8"?>
<xccdf:Tailoring xmlns:xccdf="" id="xccdf_scap-workbench_tailoring_default">
  <xccdf:benchmark href="/usr/share/xml/scap/ssg/content/ssg-fedora-ds.xml"/>
  <xccdf:version time="2016-05-26T14:15:02">1</xccdf:version>
  <xccdf:Profile id="xccdf_org.ssgproject.content_profile_common_customized" extends="xccdf_org.ssgproject.content_profile_common">
    <xccdf:title xmlns:xhtml="" xml:lang="en-US">Common Profile for General-Purpose Fedora Systems [CUSTOMIZED]</xccdf:title>
    <xccdf:description xmlns:xhtml="" xml:lang="en-US">This profile contains items common to general-purpose Fedora installations.</xccdf:description>
    <xccdf:select idref="xccdf_org.ssgproject.content_rule_package_aide_installed" selected="true"/>

In the example above I have created a really small tailoring file which selects one extra rule in the Fedora common profile from SCAP Security Guide. The most important part of the tailoring file are the Profiles. In our example it’s just the one xccdf_org.ssgproject.content_profile_common_customized profile. Let us copy the entire <xccdf:Profile> element into the clipboard.

If we look at a source datastream file things get a lot more complicated. There are catalogs, checklists, checks, extended components and all sorts of other things. Let us assume that our datastream only contains one XCCDF Benchmark. We first need to find it. Look for the <xccdf:Benchmark> element. Keep in mind that the XML namespace prefixes may differ depending on where you got the content.

<ds:component id="" timestamp="2016-05-10T14:08:41"><Benchmark xmlns="" id="xccdf_org.ssgproject.content_benchmark_FEDORA" resolved="1" xml:lang="en-US" style="SCAP_1.2">
  <status date="2016-05-10">draft</status>
  <title xml:lang="en-US">Guide to the Secure Configuration of Fedora</title>
  <description xml:lang="en-US">This guide presents a catalog of security-relevant configuration
settings for Fedora operating system formatted in the eXtensible Configuration
Checklist Description Format (XCCDF).
<html:br xmlns:html=""/>
<html:br xmlns:html=""/>
Providing system administrators with such guidance informs them how to securely
configure systems under their control in a variety of network roles.  Policy

OK, so we have found the Benchmark!  That’s the hardest part of this whole operation. We now need to find a good place to insert the Profile element. I like to insert tailored profiles as the last Profile in the benchmark. This ensures that the profiles they are derived from come first.

    <refine-value idref="xccdf_org.ssgproject.content_value_var_accounts_password_warn_age_login_defs" selector="7"/>
    <refine-value idref="xccdf_org.ssgproject.content_value_var_auditd_num_logs" selector="5"/>
    <refine-value idref="xccdf_org.ssgproject.content_value_sshd_idle_timeout_value" selector="5_minutes"/>
  ... INSERT HERE ...
  <Group id="xccdf_org.ssgproject.content_group_intro">
    <title xml:lang="en-US">Introduction</title>
    <description xml:lang="en-US">

Insert the Profile, make sure you add the namespace declaration if necessary, save the file and we are done! We can now upload this file to Satellite 6 and use our customized profile.

Option 2: Use a script

I have written a small Python helper script that does this entire surgical operation for you. Check it out at


./ ssg-fedora-ds.xml ssg-fedora-ds-tailoring.xml --output o.xml

It is a quick and dirty script, pull requests welcome.

The resulting file can be used in Satellite 6 and the customized profile will show up.


atomic scan and openscap-daemon

I would like to thank Brent Baude, Zbynek Moravec, Simon Lukasik, Dan Walsh and others who contributed to this feature!


Containers are a very big topic today, almost all businesses are looking into deploying their future services using containers. At the same time, container technology is transitioning from being a developer toy tool to something that businesses rely on. That means that container users are now focusing on security and reliability.

In this blog post we will discuss a new security related feature in Project Atomic that allows users to check whether their containers have known vulnerabilities. This allows the users to catch and replace containers that have vulnerabilities and thus prevent exploits.


Vulnerabilities are potentially a very costly problem for production deployments — internal or customer data leaks, fraud, … The bigger the deployment with more different containers images being used the tougher it gets to track vulnerabilities. Having a tool that can scan all containers we have deployed for vulnerabilities without affecting services would clearly help a lot.


We will need:

There are two major setups that we will discuss.

Everything on the same host (simple)


We could install all 3 parts on the host computer and then scan containers that are on that computer.

# assuming Fedora 23
dnf install atomic
dnf install openscap-daemon

systemctl enable openscap-daemon
systemctl start openscap-daemon


OpenSCAP in SPC (preferred)


We could install Atomic on the host computer, then install a super-privileged container with openscap-daemon, openscap and Atomic inside. The host Atomic will request the SPC to scan containers on the host machine.

This arrangement seems more tricky and complex but in the end is easier to manage because we can just pull the latest version of the SPC to install and/or update.

# assuming Fedora 23 and a self-built SPC
dnf install atomic
git clone
cd openscap-daemon/atomic
docker build f23_spc
# replace ID with the final ID that `docker build` gives you
atomic install $ID
atomic run $ID
# assuming Fedora 23 and a pre-built SPC


OK, now we have all the bits we need. Let’s use them.

# scanning a single container
atomic scan $ID
# scanning a single container image
atomic scan $ID
# scanning all images and all containers
atomic scan --all

Example output:

$ atomic scan 82ad5fa11820


Container/Image   Cri   Imp   Med   Low  
---------------   ---   ---   ---   ---  
82ad5fa11820      1     2     7     2  
$ atomic scan --detail 82ad5fa11820


OS : Red Hat Enterprise Linux Server release 7.1 (Maipo)
Critical : 1
CVE : RHSA-2015:1981: nss, nss-util, and nspr security update (Critical)
RHSA ID : RHSA-2015:1981-00

Important : 2
CVE : RHSA-2015:2172: glibc security update (Important)
RHSA ID : RHSA-2015:2172-00

CVE : RHSA-2015:1840: openldap security update (Important)
RHSA ID : RHSA-2015:1840-00

Moderate : 7
CVE : RHSA-2015:2199: glibc security, bug fix, and enhancement update (Moderate)
RHSA ID : RHSA-2015:2199-00

CVE : RHSA-2015:2159: curl security, bug fix, and enhancement update (Moderate)
RHSA ID : RHSA-2015:2159-00

CVE : RHSA-2015:2155: file security and bug fix update (Moderate)
RHSA ID : RHSA-2015:2155-00

CVE : RHSA-2015:2154: krb5 security, bug fix, and enhancement update (Moderate)
RHSA ID : RHSA-2015:2154-00

CVE : RHSA-2015:2131: openldap security, bug fix, and enhancement update (Moderate)
RHSA ID : RHSA-2015:2131-00

CVE : RHSA-2015:2108: cpio security and bug fix update (Moderate)
RHSA ID : RHSA-2015:2108-00

CVE : RHSA-2015:2101: python security, bug fix, and enhancement update (Moderate)
RHSA ID : RHSA-2015:2101-00

Low : 2
CVE : RHSA-2015:2140: libssh2 security and bug fix update (Low)
RHSA ID : RHSA-2015:2140-00

CVE : RHSA-2015:2111: grep security and bug fix update (Low)
RHSA ID : RHSA-2015:2111-00


We are working to get all of those parts packaged and then publish the ready-made SPC. In the future `atomic scan` may even pull it automatically so no installation other than Atomic should be required.

Further reading

Evaluate Virtual Machines for SCAP Compliance

Recently I have been working on oscap-vm — a script that allows SCAP evaluation of virtual machines and virtual machine storage images. In a way it is similar to the other OpenSCAP wrapper utilities — oscap-ssh and oscap-docker. It was merged to OpenSCAP and will be part of 1.2.7 release, so let us introduce it.

oscap-vm mounts the storage of a VM and sets the oscap tool to scan it in offline mode. That means that you can scan a virtual machine from the host without installing OpenSCAP on it — you can perform an agent-less SCAP scan. Root rights are not required, if you are permitted to access and change the VM you are permitted to run oscap-vm on it. The virtual machine storage is mounted read-only, there is no risk of damage to the filesystem. Because of this you cannot automatically remediate a VM using this tool. If you want remediation functionality for virtual machines and containers, please tell us.

Both XCCDF and OVAL evaluation are supported. You can use plain XCCDF files, source datastreams or plain OVAL files.
Let’s go over a few use-cases. I am using a virtual machine called rhel7.2 in the following examples.

Evaluate a running VM

$ oscap-vm domain rhel7.2 xccdf eval --profile xccdf_org.ssgproject.content_profile_stig-rhel7-server-upstream /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml 
Mounting guestfs domain 'rhel7.2' to '/tmp/tmp.c69yOdlBNZ'...
Title   Encrypt Partitions
Rule    xccdf_org.ssgproject.content_rule_encrypt_partitions
Ident   CCE-27128-8
Result  notchecked


Title   Create Warning Banners for All FTP Users
Rule    xccdf_org.ssgproject.content_rule_ftp_present_banner
Result  pass

Unmounting '/tmp/tmp.c69yOdlBNZ'...

Evaluate a storage image

$ oscap-vm image /var/lib/libvirt/images/rhel7.2.qcow2 xccdf eval --profile xccdf_org.ssgproject.content_profile_stig-rhel7-server-upstream /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml 
Mounting guestfs image '/var/lib/libvirt/images/rhel7.2.qcow2' to '/tmp/tmp.PgfWcB0R4g'...
Title   Encrypt Partitions
Rule    xccdf_org.ssgproject.content_rule_encrypt_partitions
Ident   CCE-27128-8
Result  notchecked


Title   Enable SSH Warning Banner
Rule    xccdf_org.ssgproject.content_rule_sshd_enable_warning_banner
Ident   CCE-27314-4
Result  fail

Title   Create Warning Banners for All FTP Users
Rule    xccdf_org.ssgproject.content_rule_ftp_present_banner
Result  pass

Unmounting '/tmp/tmp.PgfWcB0R4g'...

Check VM for CVE vulnerabilities

$ wget
$ oscap-vm domain rhel7.2 oval eval Red_Hat_Enterprise_Linux_7.xml 
Mounting guestfs domain 'rhel7.2' to '/tmp/tmp.NbvfmaKHbZ'...
Definition oval:com.redhat.rhsa:def:20151852: false
Definition oval:com.redhat.rhsa:def:20151840: false
Definition oval:com.redhat.rhsa:def:20151834: false
Definition oval:com.redhat.rhsa:def:20151793: false
Definition oval:com.redhat.rhsa:def:20140685: false
Definition oval:com.redhat.rhsa:def:20140684: false
Definition oval:com.redhat.rhsa:def:20140680: false
Definition oval:com.redhat.rhsa:def:20140679: false
Definition oval:com.redhat.rhsa:def:20140678: false
Definition oval:com.redhat.rhsa:def:20140675: false
Evaluation done.
Unmounting '/tmp/tmp.NbvfmaKHbZ'...

SCAP Security Guide now has an HTML guide for each profile

In the past the SCAP Security Guide project built one or just a few HTML guides for some chosen profiles. The build system also used a special profile called allrules which is no longer supported since OpenSCAP 1.1.0. This caused issues when building SSG against new versions of OpenSCAP.

To fix it once and for all I have created a pull request that has been merged and has been released as part of SSG 0.1.24. Since then SSG builds one guide for each profile and provides an index file that allows user to switch between the profiles. The reasoning for building so many guides is that if we ship a profile it is important enough to warrant shipping an HTML guide for it as well.

Here is how the new profile switcher looks like:


I have uploaded the RHEL6 guides with the profile switcher to fedorapeople, take a look:

We plan to install these guides in downstream packages (e.g. Fedora). Instead of bundling them with the main package we will create a subpackage -doc with the guides and other optional documentation material.

Feedback appreciated! Do you think we should upload these guides somewhere for people to browse? Are there any features missing?