As I contribute more and more patches to SCAP Security Guide I got increasingly frustrated with the build speeds. A full SSG build with make -j 4 took 2m21.061s and that’s without any XML validation taking place. I explored a couple of options how I could cut this time significantly. I started by profiling the Makefile and found that a massive amount of time is spent on 2 things.
Generating HTML guides
We generate a lot of HTML guides as part of SSG builds and we do that over and over for each profile of each product. That’s a lot of HTML guides in total. Generating one HTML guide (namely the RHEL7 PCI-DSS profile from the datastream) took over 3 seconds on my machine. While not a huge number this adds up to a long time with all the guides we are generating. Optimizing HTML guides the first thing I focused on.
I found that we are often selecting huge nodesets over and over instead of reusing them. Fixing this brought the times down roughly 30%. I found a couple other inefficiencies and was able to save an additional 5-10% there. Overall I have optimized it roughly 35-40% in common cases.
During the optimization I have accidentally fixed a pretty jarring bug regarding refine-value and value selectors. We used to select a big nodeset of all cdf:Value elements in the entire document, then select all their cdf:values inside and choose the last based on the selector. This is clearly wrong because we need to select the right cdf:Value with the right ID and then look at only its selectors. Fixing that make the transformation faster as well because the right cdf:Value was already pre-selected.
Old XSLTs:
$ time ../../../shared/utils/build-all-guides.py -j 1 --input ssg-rhel7-ds.xml real 0m16.736s user 0m16.349s sys 0m0.397s
New XSLTs:
$ time ../../../shared/utils/build-all-guides.py -j 1 --input ssg-rhel7-ds.xml real 0m11.203s user 0m10.836s sys 0m0.379s
EDIT: I found more optimization opportunities, latest data as of 2016-08-10:
real 0m3.399s user 0m2.986s sys 0m0.409s
I won’t be redoing the entire test-suite and all the graphs but the final savings are much better than it shows in the graph. Generating all RHEL7 SDS guides takes less than 2 seconds on my machine after the optimizations.
Transforming XCCDF 1.1 to 1.2
It took 30 seconds on my machine to transform RHEL6 XCCDF 1.1 to 1.2. That is just way too much for a simple operation like that. Clearly something was wrong with the XSLT transformation. As soon as I profiled the XSLT using xsltproc --profile I found that we select the entire DOM over and over for every @idref in the tree. That is just silly. I fixed that by using xsl:key and using the very same @idref to element mapping for all lookups. This saved a lot of time.
Doing the RHEL6 XCCDF 1.1 to 1.2 transformation with old XSLTs real 0m34.635s user 0m34.585s sys 0m0.047s Doing the RHEL6 XCCDF 1.1 to 1.2 transformation with new XSLTs real 0m0.619s user 0m0.573s sys 0m0.045s
The numbers were similar for the RHEL7 XCCDF 1.1 to 1.2 transformation.
Final results for the SSG build
I started with 2m21.061s and my goal was to bring that down to 50%. The final time on my machine after the optimizations with make -j 4 is 1m4.217s. Savings of roughly 55%. Most of those savings are in the XCCDF 1.1 to 1.2 transformation that we do for every product.
The savings are great on my beefy work laptop (i7-5600U) but we should benefit even more from them on our Jenkins slaves that aren’t as powerful. I have yet to test how much they would help there but I estimate it will be 10 minutes for each build.
Correctness
When I suggested to deploy these improvements on our Jenkins slaves, Jan Lieskovsky brought up an important point about correctness. We decided to diff old and new guides and old and new XCCDF 1.2s to be sure we aren’t changing behavior. Please see the attached ZIP file for a test case I created to verify that we haven’t changed behavior. During the process of creating this test case I discovered that I have accidentally fixed a bug mentioned above 🙂 To silence the diffs I have introduced just this bug into the new XSLTs I used. This made the performance slightly worse so keep that in mind when looking at the numbers.
./test_xccdf11_to_12.sh Doing the RHEL6 XCCDF 1.1 to 1.2 transformation with old XSLTs real 0m34.635s user 0m34.585s sys 0m0.047s Doing the RHEL6 XCCDF 1.1 to 1.2 transformation with new XSLTs real 0m0.619s user 0m0.573s sys 0m0.045s Diffing old_xslt_output/ssg-rhel6-xccdf-1.2.xml and new_xslt_output/ssg-rhel6-xccdf-1.2.xml The files are the same. Doing the RHEL7 XCCDF 1.1 to 1.2 transformation with old XSLTs real 0m33.146s user 0m33.089s sys 0m0.050s Doing the RHEL7 XCCDF 1.1 to 1.2 transformation with new XSLTs real 0m0.749s user 0m0.702s sys 0m0.047s Diffing old_xslt_output/ssg-rhel7-xccdf-1.2.xml and new_xslt_output/ssg-rhel7-xccdf-1.2.xml The files are the same.
./test_html_guides.sh Doing the RHEL6 and 7 SDS HTML guide transformations with old XSLTs real 0m39.104s user 0m38.605s sys 0m0.491s Doing the RHEL6 and 7 SDS HTML guide transformations with new XSLTs real 0m28.974s user 0m28.531s sys 0m0.433s Diffing old_xslt_output/guides_for_diff and new_xslt_output/guides_for_diff No differences.
UPDATE: Jenkins build times (2016-08-12)
Here is a graph of Jenkins build times, you can see how the build times gradually went lower as optimizations got onto the Jenkins slaves. There are occasional build time spikes caused by load when multiple pull requests were submitted at once but overall the performance has been improved.