Tag Archives: scap

OpenSCAP XSLT performance improvements for faster SSG builds

As I contribute more and more patches to SCAP Security Guide I got increasingly frustrated with the build speeds. A full SSG build with make -j 4 took 2m21.061s and that’s without any XML validation taking place. I explored a couple of options how I could cut this time significantly. I started by profiling the Makefile and found that a massive amount of time is spent on 2 things.

Generating HTML guides

xslt_optimization_html_guide_chart

We generate a lot of HTML guides as part of SSG builds and we do that over and over for each profile of each product. That’s a lot of HTML guides in total. Generating one HTML guide (namely the RHEL7 PCI-DSS profile from the datastream) took over 3 seconds on my machine. While not a huge number this adds up to a long time with all the guides we are generating. Optimizing HTML guides the first thing I focused on.

I found that we are often selecting huge nodesets over and over instead of reusing them. Fixing this brought the times down roughly 30%. I found a couple other inefficiencies and was able to save an additional 5-10% there. Overall I have optimized it roughly 35-40% in common cases.

During the optimization I have accidentally fixed a pretty jarring bug regarding refine-value and value selectors. We used to select a big nodeset of all cdf:Value elements in the entire document, then select all their cdf:values inside and choose the last based on the selector. This is clearly wrong because we need to select the right cdf:Value with the right ID and then look at only its selectors. Fixing that make the transformation faster as well because the right cdf:Value was already pre-selected.

Old XSLTs:

$ time ../../../shared/utils/build-all-guides.py -j 1 --input ssg-rhel7-ds.xml
real 0m16.736s
user 0m16.349s
sys  0m0.397s

New XSLTs:

$ time ../../../shared/utils/build-all-guides.py -j 1 --input ssg-rhel7-ds.xml
real 0m11.203s
user 0m10.836s
sys  0m0.379s

EDIT: I found more optimization opportunities, latest data as of 2016-08-10:

real 0m3.399s
user 0m2.986s
sys  0m0.409s

I won’t be redoing the entire test-suite and all the graphs but the final savings are much better than it shows in the graph. Generating all RHEL7 SDS guides takes less than 2 seconds on my machine after the optimizations.

Transforming XCCDF 1.1 to 1.2

xslt_optimization_xccdf11_12_chart

It took 30 seconds on my machine to transform RHEL6 XCCDF 1.1 to 1.2. That is just way too much for a simple operation like that. Clearly something was wrong with the XSLT transformation. As soon as I profiled the XSLT using xsltproc --profile I found that we select the entire DOM over and over for every @idref in the tree. That is just silly. I fixed that by using xsl:key and using the very same @idref to element mapping for all lookups. This saved a lot of time.

Doing the RHEL6 XCCDF 1.1 to 1.2 transformation with old XSLTs

real 0m34.635s
user 0m34.585s
sys  0m0.047s

Doing the RHEL6 XCCDF 1.1 to 1.2 transformation with new XSLTs

real 0m0.619s
user 0m0.573s
sys  0m0.045s

The numbers were similar for the RHEL7 XCCDF 1.1 to 1.2 transformation.

Final results for the SSG build

I started with 2m21.061s and my goal was to bring that down to 50%. The final time on my machine after the optimizations with make -j 4 is 1m4.217s. Savings of roughly 55%. Most of those savings are in the XCCDF 1.1 to 1.2 transformation that we do for every product.

The savings are great on my beefy work laptop (i7-5600U) but we should benefit even more from them on our Jenkins slaves that aren’t as powerful. I have yet to test how much they would help there but I estimate it will be 10 minutes for each build.

Correctness

When I suggested to deploy these improvements on our Jenkins slaves, Jan Lieskovsky brought up an important point about correctness. We decided to diff old and new guides and old and new XCCDF 1.2s to be sure we aren’t changing behavior. Please see the attached ZIP file for a test case I created to verify that we haven’t changed behavior. During the process of creating this test case I discovered that I have accidentally fixed a bug mentioned above 🙂 To silence the diffs I have introduced just this bug into the new XSLTs I used. This made the performance slightly worse so keep that in mind when looking at the numbers.

./test_xccdf11_to_12.sh 
Doing the RHEL6 XCCDF 1.1 to 1.2 transformation with old XSLTs

real 0m34.635s
user 0m34.585s
sys  0m0.047s

Doing the RHEL6 XCCDF 1.1 to 1.2 transformation with new XSLTs

real 0m0.619s
user 0m0.573s
sys  0m0.045s

Diffing old_xslt_output/ssg-rhel6-xccdf-1.2.xml and new_xslt_output/ssg-rhel6-xccdf-1.2.xml
The files are the same.


Doing the RHEL7 XCCDF 1.1 to 1.2 transformation with old XSLTs

real 0m33.146s
user 0m33.089s
sys  0m0.050s

Doing the RHEL7 XCCDF 1.1 to 1.2 transformation with new XSLTs

real 0m0.749s
user 0m0.702s
sys  0m0.047s

Diffing old_xslt_output/ssg-rhel7-xccdf-1.2.xml and new_xslt_output/ssg-rhel7-xccdf-1.2.xml
The files are the same.
./test_html_guides.sh 
Doing the RHEL6 and 7 SDS HTML guide transformations with old XSLTs

real 0m39.104s
user 0m38.605s
sys  0m0.491s

Doing the RHEL6 and 7 SDS HTML guide transformations with new XSLTs

real 0m28.974s
user 0m28.531s
sys  0m0.433s

Diffing old_xslt_output/guides_for_diff and new_xslt_output/guides_for_diff
No differences.

UPDATE: Jenkins build times (2016-08-12)

xslt_optimization_ssg_jenkins_build_times
Here is a graph of Jenkins build times, you can see how the build times gradually went lower as optimizations got onto the Jenkins slaves. There are occasional build time spikes caused by load when multiple pull requests were submitted at once but overall the performance has been improved.

SCAP Workbench 1.1.0

The new SCAP Workbench is out! This is the biggest release to date. We focused on improving the typical use-case of tailoring and remote scanning. This is also the first release to have Windows and MacOS X support!

Update: Thanks to the great work of Jakub Jelen, we now have a testing release for Windows that supports remote scan! Give it a try, report any issues: scap-workbench-1.1.0-win32-remote-scan-testing.zip. SCAP Workbench 1.1.1 will be released some time soon and will have remote scanning from Windows included.

Fedora updates for F22, F21 and F20 are pending. Testing and karma would be greatly appreciated! This release brings so many fixes and does not break existing use-cases that I decided to push it to older Fedoras as well. Even though it is a major release.

Screenshots

sw-1.1.0_1 sw-1.1.0_2

sw-1.1.0_3 sw-1.1.0_4

What’s new?

1.1.0 is a packed major release, the number of changes is second only to the 0.8.0 C++ rewrite.

  • Windows support – including a native MSI installer
  • MacOS X support – including a native dmg image
  • Complete redesign of the main window, with rich-text rule descriptions
  • Better SCAP Security Guide integration
  • Tailoring window greatly improved – shows relationships between values and rules
  • Opens bzip2 files
  • Performance improvements when loading big SCAP files
  • Countless UX improvements
  • And a lot more, a total of 49 tickets fixed, plus fixes merged from maintenance branches

Where to report issues?

The best place to report issues is the trac bug tracker. However I also accept reports via the mailing list or even comments to this blog post.

Waivers in openscap HTML report

XCCDF supports waivers by the means of the cdf:override element. Support for it in the openscap API has recently been greatly enhanced by Šimon Lukašík. Therefore I have looked into adding support for waivers in the HTML report.

This blog post talks about openscap master as of commit 30f9a224bc25f2127462d30ec1e4c0d499a23511.

We decided to use the term waiver instead of override. The situation looks similar to tailoring vs customization to us. Waiver should be understood by more people without even looking at any documentation.

Rule overview now shows a small label next to waived rules. This label signifies that this particular rule has at least one waiver.

waiver1

When you click on the waived rule to see the details you will be presented with description of the waiver including information on who has performed it and when. If there are more waivers they are all shown in the order they appear in the original XML file.

waiver2

I know this feature has been requested by the community for a long time so I would like to gather some feedback to get it as right as possible.

Please note that the HTML reports themselves don’t allow you to interactively perform waivers, they just show them. As of now I don’t have plans to support interactive waivers but we may implement something like that in the future.

EDIT: I have implemented a prototype that allows interactive waivers using JavaScript. The waivers are not committed yet but I plan to commit them to master next week. Take a look at https://mpreisle.fedorapeople.org/openscap/interactive_waiver.html

Suggestions welcome!

EDIT2: Based on the feedback we decided not to include interactive waiver in the openscap reports themselves. Instead I will add HTML elements to make it easier to implement it in openscap integrations. The patch of the prototype follows and can be cleanly applied on top of 30f9a224bc25f2127462d30ec1e4c0d499a23511.

diff --git a/xsl/xccdf-report-impl.xsl b/xsl/xccdf-report-impl.xsl
index 08c50cf..f7c26f5 100644
--- a/xsl/xccdf-report-impl.xsl
+++ b/xsl/xccdf-report-impl.xsl
@@ -293,9 +293,7 @@ Authors:
 <xsl:with-param name="profile" select="$profile"/>
 </xsl:call-template>
 </a>
- <xsl:if test="$ruleresult/cdf:override">
- &#160;<span class="label label-warning">waived</span>
- </xsl:if>
+ &#160;<span class="label label-warning waiver-label"><xsl:if test="not($ruleresult/cdf:override)"><xsl:attribute name="style">display: none</xsl:attribute></xsl:if>waived</span>
 </td>
 <td style="text-align: center"><xsl:value-of select="$ruleresult/@severity"/></td>
 <td class="rule-result rule-result-{$result}">
@@ -304,7 +302,7 @@ Authors:
 <xsl:with-param name="ruleresult" select="$result"/>
 </xsl:call-template>
 </xsl:variable>
- <div>
+ <div class="inner">
 <abbr title="{$result_tooltip}"><xsl:value-of select="$result"/></abbr>
 </div>
 </td>
@@ -606,7 +604,8 @@ Authors:
 <div class="panel-body">
 <table class="table table-striped table-bordered">
 <tbody>
- <tr><td class="col-md-3">Rule ID</td><td class="rule-id col-md-9"><xsl:value-of select="$item/@id"/></td></tr>
+ <xsl:variable name="itemid" select="$item/@id"/>
+ <tr><td class="col-md-3">Rule ID</td><td class="rule-id col-md-9"><xsl:value-of select="$itemid"/></td></tr>
 <tr><td>Result</td>
 <td class="rule-result rule-result-{$result}">
 <xsl:variable name="result_tooltip">
@@ -614,9 +613,17 @@ Authors:
 <xsl:with-param name="ruleresult" select="$result"/>
 </xsl:call-template>
 </xsl:variable>
- <div>
+ <div class="inner">
 <abbr title="{$result_tooltip}"><xsl:value-of select="$result"/></abbr>
 </div>
+ <!-- see openscap.js for clues about the following 2 divs, they are used
+ for interactive waivers. -->
+ <div class="js-only waiver-button">
+ <xsl:if test="$result != 'pass' and $result != 'fixed' and $result != 'notselected'">
+ <button class="waive-button btn btn-warning btn-lg" type="button" onclick="return showWaiverForm($(this).closest('.rule-detail'), '{$itemid}')">Waive</button>
+ </xsl:if>
+ </div>
+ <div class="js-only waiver-form"></div>
 </td></tr>
 <tr><td>Time</td><td><xsl:value-of select="$ruleresult/@time"/></td></tr>
 <tr><td>Severity</td><td><xsl:value-of select="$ruleresult/@severity"/></td></tr>
@@ -628,8 +635,8 @@ Authors:
 <xsl:with-param name="item" select="$item"/>
 </xsl:call-template>
 </td></tr>
- <xsl:if test="$ruleresult/cdf:override">
- <tr><td colspan="2">
+ <tr><td colspan="2" class="waivers">
+ <xsl:if test="$ruleresult/cdf:override">
 <xsl:for-each select="$ruleresult/cdf:override">
 <xsl:variable name="old-result" select="cdf:old-result/text()"/>
 
@@ -643,8 +650,8 @@ Authors:
 </small>
 </div>
 </xsl:for-each>
- </td></tr>
- </xsl:if>
+ </xsl:if>
+ </td></tr>
 <tr><td colspan="2"><div class="description">
 <p>
 <xsl:apply-templates mode="sub-testresult" select="$item/cdf:description">
diff --git a/xsl/xccdf-resources-build.sh b/xsl/xccdf-resources-build.sh
index ec5f584..089d38a 100755
--- a/xsl/xccdf-resources-build.sh
+++ b/xsl/xccdf-resources-build.sh
@@ -10,12 +10,15 @@ cat xccdf-resources/jquery.treetable.css >> $ALL_CSS
 cat xccdf-resources/jquery.treetable.theme.css >> $ALL_CSS
 cat xccdf-resources/openscap.css >> $ALL_CSS
 csstidy $ALL_CSS --template=highest $ALL_CSS_MIN
+#cp $ALL_CSS $ALL_CSS_MIN
 rm $ALL_CSS
+
 echo "" > $ALL_JS
 cat xccdf-resources/jquery.treetable.js >> $ALL_JS
 cat xccdf-resources/bootstrap.min.js >> $ALL_JS
 cat xccdf-resources/openscap.js >> $ALL_JS
 slimit $ALL_JS > $ALL_JS_MIN
+#cp $ALL_JS $ALL_JS_MIN
 rm $ALL_JS
 
 XCCDF_RESOURCES="xccdf-resources.xsl"
diff --git a/xsl/xccdf-resources/openscap.css b/xsl/xccdf-resources/openscap.css
index 2e0f8e4..6521c84 100644
--- a/xsl/xccdf-resources/openscap.css
+++ b/xsl/xccdf-resources/openscap.css
@@ -1,11 +1,13 @@
 tr.rule-overview-needs-attention td a { color: #d9534f }
 
-td.rule-result div, span.rule-result { text-align: center; font-weight: bold; color: #fff; background: #808080 }
-td.rule-result-fail div, span.rule-result-fail { background: #d9534f }
-td.rule-result-error div, span.rule-result-error { background: #d9534f }
-td.rule-result-unknown div, span.rule-result-unknown { background: #f0ad4e }
-td.rule-result-pass div, span.rule-result-pass { background: #5cb85c }
-td.rule-result-fixed div, span.rule-result-fixed { background: #5cb85c }
+td.rule-result div.inner, span.rule-result { text-align: center; font-weight: bold; color: #fff; background: #808080 }
+td.rule-result-fail div.inner, span.rule-result-fail { background: #d9534f }
+td.rule-result-error div.inner, span.rule-result-error { background: #d9534f }
+td.rule-result-unknown div.inner, span.rule-result-unknown { background: #f0ad4e }
+td.rule-result-pass div.inner, span.rule-result-pass { background: #5cb85c }
+td.rule-result-fixed div.inner, span.rule-result-fixed { background: #5cb85c }
+
+td.rule-result div.waiver-button { float: right }
 
 .js-only { display: none }
 
diff --git a/xsl/xccdf-resources/openscap.js b/xsl/xccdf-resources/openscap.js
index f76645b..70e87b3 100644
--- a/xsl/xccdf-resources/openscap.js
+++ b/xsl/xccdf-resources/openscap.js
@@ -9,6 +9,7 @@ function openRuleDetailsDialog(rule_result_id)
 
 var clone = $("#rule-detail-" + rule_result_id).clone();
 clone.attr("id", "");
+ clone.data("idm_id", "rule-detail-" + rule_result_id);
 clone.children(".panel-heading").append(closebutton);
 closebutton.css( { "float" : "right" } );
 closebutton.css( { "margin-top" : "-=20px" } );
@@ -108,7 +109,122 @@ function ruleSearch()
 $("#search-matches").html("No rules match your search criteria!");
 }
 
-$(document).ready( function() {
+waiverCallback = function(rule_id, authority, datetime, new_result, remark)
+{
+ //alert(rule_id);
+ //alert(datetime);
+ //alert(new_result);
+ //alert(remark);
+
+ return "";
+}
+
+if (typeof waiverCallback === "undefined")
+ waiverCallback = null;
+
+if (typeof waiverDefaultAuthority === "undefined")
+ waiverDefaultAuthority = "Undefined Authority";
+
+function injectNewWaiver(rule_detail, rule_id, authority, datetime, new_result, remark)
+{
+ var previous_result = rule_detail.find(".rule-result .inner abbr").html();
+
+ rule_detail.removeClass();
+ rule_detail.addClass("panel panel-default rule-detail rule-detail-" + new_result);
+
+ var waiver_div = rule_detail.find("div.waiver-button");
+ waiver_div.hide();
+
+ var rule_result = rule_detail.find("td.rule-result");
+ rule_result.removeClass();
+ rule_result.addClass("rule-result");
+ rule_result.addClass("rule-result-" + new_result);
+ rule_result.find(".inner").html("<abbr>" + new_result + "</abbr>");
+
+ var waivers_div = rule_detail.find("td.waivers");
+ var new_waiver_div = $('<div class="alert alert-warning waiver">This rule has been waived by <strong>' + authority + '</strong> at <strong>' + datetime + '</strong>.<blockquote>' + remark + '</blockquote><small>The previous result was <span class="rule-result rule-result-' + previous_result + '">&#160;' + previous_result + '&#160;</span>.</small></div>');
+
+ waivers_div.append(new_waiver_div);
+
+ if (rule_detail.data("idm_id"))
+ {
+ rule_detail_lookup = $("#" + rule_detail.data("idm_id"));
+ injectNewWaiver(rule_detail_lookup, rule_id, authority, datetime, new_result, remark);
+ return;
+ }
+
+ var idm_base = $(rule_detail).attr("id").substring(12);
+ var rule_overview_leaf = $("#rule-overview-leaf-" + idm_base);
+
+ rule_overview_leaf.removeClass();
+ rule_overview_leaf.addClass("rule-overview-leaf rule-overview-leaf-" + new_result);
+ rule_overview_leaf.find(".waiver-label").show();
+
+ var rule_result_overview = rule_overview_leaf.find(".rule-result");
+ rule_result_overview.removeClass();
+ rule_result_overview.addClass("rule-result rule-result-" + new_result);
+ rule_result_overview.find("div").html("<abbr>" + new_result + "</abbr>");
+}
+
+function submitWaiverForm(rule_detail, rule_id, waiver_form)
+{
+ var now = new Date();
+ var datetime = now.toISOString();
+ var authority = waiver_form.find(".waiver-authority").val();
+ var new_result = waiver_form.find(".waiver-new-result option:selected").val();
+ var remark = waiver_form.find(".waiver-remark").val();
+
+ var result = waiverCallback(rule_id, authority, datetime, new_result, remark);
+ if (result == "")
+ {
+ hideWaiverForm(rule_detail, rule_id);
+ injectNewWaiver(rule_detail, rule_id, authority, datetime, new_result, remark);
+ }
+ else
+ {
+ // TODO: Show error
+ }
+
+ return false;
+}
+
+function showWaiverForm(rule_detail, rule_id)
+{
+ var waiver_div = rule_detail.find("div.waiver-button");
+ waiver_div.hide();
+ var waiver_form_div = rule_detail.find("div.waiver-form");
+
+ var waiver_form = $('<form role="form"/>');
+ var authority = $('<div class="form-group"><label class="control-label">Authority</label><input type="text" class="waiver-authority form-control" value="' + waiverDefaultAuthority + '"/></div>');
+ waiver_form.append(authority);
+ var new_result = $('<div class="form-group"><label class="control-label">New result</label><select class="waiver-new-result form-control"><option>pass</option><option>notapplicable</option></select></div>');
+ waiver_form.append(new_result);
+ var remark = $('<div class="form-group"><label class="control-label">Remark</label><textarea class="waiver-remark form-control"/></div>');
+ waiver_form.append(remark);
+ var ok_button = $('<button type="button" class="btn btn-primary btn-sm">Confirm</button>');
+ ok_button.click(function(){
+ return submitWaiverForm(rule_detail, rule_id, waiver_form);
+ });
+ waiver_form.append(ok_button);
+ var cancel_button = $('<button type="button" class="btn btn-default btn-sm">Cancel</button>');
+ cancel_button.click(function(){
+ return hideWaiverForm(rule_detail, rule_id);
+ });
+ waiver_form.append(cancel_button);
+ waiver_form_div.append(waiver_form);
+}
+
+function hideWaiverForm(rule_detail, rule_id)
+{
+ var waiver_div = rule_detail.find("div.waiver-button");
+ waiver_div.show();
+ var waiver_form_div = rule_detail.find("div.waiver-form");
+ waiver_form_div.empty();
+
+ return false;
+}
+
+$(document).ready(function(){
 $("#result-details").hide();
 $(".js-only").show();
 $(".toggle-rule-display").each(function(){

Don’t forget to run xccdf-build-resources.sh after applying the patch!