bash

How to sign XML documents using XMLDSig (XML Signature)

Tagged xmldsig, xml, signature, wtf, hl7fi, kanta, xmlsec1  Languages xml, bash

Install xmlsec1

sudo apt-get install xmlsec1

Create document

<?xml version="1.0" encoding="UTF-8"?>
<document>
  <hello>All XML is doomed to fail.</hello>
  <!-- Signature contains the signature definition -->
  <Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
    <SignedInfo>
      <CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
      <SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
      <Reference>
        <Transforms>
          <Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
          <Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
        </Transforms>
        <DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
        <DigestValue />
      </Reference>
      </SignedInfo>
    <SignatureValue />
    <KeyInfo>
      <X509Data />
    </KeyInfo>
  </Signature>
</document>

Sign document

xmlsec1 --sign --privkey-pem xxx.com.key,xxx.com.cer --output signed.xml tosign.xml

This example uses test certificates issued by VRK.

Verify document

xmlsec1 --verify --trusted-pem vrkthsp.pem --trusted-pem vrktestc.pem signed.xml

Note that a concatenated PEM file, i.e. cat vrkthsp.pem vrktestc.pem > concat.pem, does not work with xmlsec1.

How to specify which elements to sign with ds:Reference

Add one or more ds:Reference elements to specify which elements should be signed. Each element should have a unique ID in the URI attribute. The ID should be prefixed with a hash, e.g., #your-id:

<ds:Reference URI="#secret-xml-sauce">

Make sure your document contains an element having the exact ID without the hash prefix:

<Dog ID="secret-xml-sauce" name="Christian" />

Next, use the “—id-attr” switch to specify the element and attribute name:

xmlsec1 --sign --privkey-pem signing.key,signing.pem --id-attr:ID Dog --id-attr:ID structuredBody --output signed.xml tosign.xml

Note that “id” is the default attribute name. You only need —id-attr switch if you have the ID in an attribute having a different name.

How to sign multiple elements

Just add another “—id-attr: ” switch:

xmlsec1 --sign --privkey-pem signing.key,signing.pem --id-attr:ID signatureTimestamp --id-attr:ID structuredBody --output signed.xml tosign.xml

Then add another element having the given ID.

Troubleshooting

  • This error means you don’t have the correct trusted pem
certificate issuer check failed:err=2;msg=unable to get issuer certificate;issuer=/C=FI/ST=Finland/O=Vaestorekisterikeskus TEST/OU=Certification Authority Services/OU=Varmennepalvelut/CN=VRK TEST Root CA

To fix the error, take a hard look at the Issuer and Subject of all certificates in the certificate chain. For example:

openssl x509 -inform DER -in vrktestc.crt -text | grep "Issuer\|Subject"
  • xmlsec1 fails to find element containing ID
func=xmlSecXPathDataExecute:file=xpath.c:line=273:obj=unknown:subj=xmlXPtrEval:error=5:libxml2 library function failed:expr=xpointer(id('ID_OF_ELEMENT_TO_SIGN'))

Note that the XPATH queries are case sensitive. This means you might have to specify both the name of the element and the name of the ID attribute like this:

xmlsec1 sign --id-attr:ID elementThatYouWantToSign ...

For more solutions to issues, see sgros.blogspot.com: http://sgros.blogspot.com/2013/01/signing-xml-document-using-xmlsec1.html

Bulk renaming of files

Tagged rename, regexp, bulk, filename, bash, perl, linux  Languages bash

Rename the files in a directory by replacing a space with an underscore. The rename program comes with most modern Linux distros.

rename 's/\ /_/g' *.*

Create an flv movie

Tagged flv, movie, audio, ffmpeg, flash  Languages bash

Create an flv movie that is 320 wide by 240 high. For audio functionality ffmpeg needs to be compiled with the following parameters:

--enable-amr_nb 
--enable-amr_nb-fixed 
--enable-amr_wb 
--enable-mp3lame

The snippet itself:

ffmpeg -i mymovie.avi -s 320x240 -ar 44100 -r 12 mymovie.flv

Add duration metadata into flv movie

Tagged flv, movie, flash, duration, metadata, flvtool2  Languages bash

By default an flv movie doesn't contain the duration metadata. Using the flvtool2 program it is injected like this into the movie file.

cat mymovie.flv | flvtool2 -U stdin mymovie.flv

Create a thumbnail of a movie.

Tagged thumbnail, movie, ffmpeg, jpg  Languages bash

Create a thumbnail that is 320 wide by 240 high and five seconds into the movie.

ffmpeg -i mymovie.avi -f mjpeg -t 0.001 -ss 5 -s "320x240" mymovie.jpg

Colorize grep

Tagged grep, bash, color, linux  Languages bash

Put this export in .bashrc to make grep colorize the text it found.

export GREP_OPTIONS='--color=auto'

SSH tunneling

Tagged ssh, tunneling, port, secure  Languages bash

SSH tunneling to bypass overly strict firewalls for services you need. The first port is that of the remote service, and the latter is the port you want to use on the local computer. For clarity I usually use the same port on both computers if possible. Here we initiate a tunnel into the vnc server of a remote host.

ssh -L 5905:localhost:5905 marko@remotehost

Next we open the vnc viewer into localhost, which in fact is the tunnel into the remote host.

vncviewer localhost:5

Enable remote debugging in weblogic

Tagged debug, weblogic  Languages bash

Sometimes it's necessary to debug because you can't write a test for it (e.g a legacy system). Just add the following parameters to the server startup and connect your preferred debugger into port 1044 (or whatever you choose the port to be). Works in weblogic, but should work in JBoss and other java based application servers too.

-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=1044

Delete messages containing a keyword in mutt

Tagged mutt, mail, bulk, delete  Languages bash

Bulk operations in mutt, a console based email client, are easy. Here's how to delete messages containing the keyword 'Newsletter'.

T
Newsletter
;
d
$

T - asks which messages you want to tag and you reply with 'Newsletter'.

; - asks which action you want to run on the tagged messages.

d - tells that the action you want is delete.

$ - synchronizes the view with the underlying persistence layer.

Creating a local Debian mirror for your Xen servers

Tagged xen, debian, etch, local mirror, amd64, anonftpsync, nginx  Languages bash

Once you've bought a dual or quad Xeon and started to experiment with virtualization you will soon want to create your local mirror to make installs lightning fast. This is a step-by-step how i did it.

First create the Xen that will be our mirror server. The size requirements can be found here: Debian mirror sizes The combined size of amd64 architecture and architecture independent files was 39Gb on 1.9.2007. So I made the image 50Gb big. Remember to change this mirror to a location near you.

xen-create-image --hostname=mirrors.aktagon.com \
--size=50Gb --swap=256Mb --ip=10.0.0.44 \
--netmask=255.255.255.0 --gateway=10.0.0.2 \
--force --dir=/work/vserver --memory=256Mb \
--arch=amd64 \
--kernel=/boot/vmlinuz-2.6.18-5-xen-amd64 \
--debootstrap --dist=etch \
--mirror= http://ftp.fi.debian.org/debian/\
--passwd

Then ssh into your new Xen as root.

ssh -l root mirrors.aktagon.com

Make base configurations for a fresh Xen.

apt-get update && apt-get install locales console-data && dpkg-reconfigure locales

Then get the mirror synchronization script from Debian.

wget "http://www.debian.org/mirror/anonftpsync"
chmod a+x anonftpsync

Then install dependencies for anonftpsync script. Otherwise the script will fail with a -bash: lockfile: command not found error.

apt-get install procmail

Install nginx.

apt-get install nginx

Configure anonftpsync with your favorite editor and change the lines below. These settings will setup a mirror only for amd64 files. You could remove i386 from the excluded architectures, but then a 50Gb image won't fit all the files.

TO=/var/www/debian
RSYNC_HOST=ftp.fi.debian.org
RSYNC_DIR=debian
LOGDIR=/var/log/mirroring
ARCH_EXCLUDE="alpha arm hppa hurd-i386 i386 ia64 m68k mipsel mips powerpc s390 sh sparc source"

Make the log directory.

mkdir -p /var/log/mirroring

Configure nginx by modifying /etc/nginx/nginx.conf with your favorite editor. Just add the autoindex line into server { location / { context

# abbreviated start of file for clarity...
    server {
        listen       80;
        server_name  localhost;

        access_log  /var/log/nginx/localhost.access.log;

        location / {
            root   /var/www;
            # add the line below to allow directory listing
            autoindex  on;
            index  index.html index.htm;
        }
   # abbreviated end of file for clarity...

Do the synchronizing. And wait... for a long while. On a 8/1Mbit cable the first synchronize took roughly 20 hours.

./anonftpsync

Now modify your /etc/apt/sources.list on existing Xen images to use your local mirror. And remember to create new Xen images using your new mirror :) In the above case the URL is http://mirrors.aktagon.com/debian

NB: there is no public mirrors.aktagon.com available... sorry.