Archive for the Code Category

Network/SQL multi host Emerge for Gentoo

Posted in Code on 09/03/2012 by Undersys

I have a lot of Gentoo hosts. Going to each one of them and running emerge xxxx what ever is really annoying.
Its also a waste of my time.  I set out to fix that.

Design goals are :-
1) Can query a SQL DB to find out what hosts need updating.
2) Can run against a single host or many hosts.
3) Can preform emerge sync, revdev- rebuld, emerge packages and tell me what needs uptating in etc.
4) Can run from a central host.
5) Can run from a crontab and generate a log file for me to inspect later.

Currently I can do this :-

./genhstutil.rb
Preform maintenance on mutiple Gentoo hosts over the network

Usage:
 [options] {host} or {package}
 where {package} is the name of a gentoo package atom
 where {host} is the dns name of a host
 where [options] are:
 --esync, -e: Run emerge sync on {host}
 --breif, -b: Get a one line package update summary on {host}
 --detail, -d: Get a detailed package update list on {host}
 --world, -w: Emerge world on {host}
 --system, -s: Emerge system on {host}
 --revdep, -r: Run revdep rebuild on {host}
 --ipkg, -i: Install {package} on a {host}
 --etcup, -t: Check a host for etc-updates
 --esync-all, -y: Run 'esync' on all Gentoo hosts
 --breif-all, -f: Run 'breif' on all Gentoo hosts
 --detail-all, -a: Run 'detail' on all Gentoo hosts
 --ipkg-all, -p: Install package on all Gentoo hosts
 --world-all, -o: Run emerge world on all Gentoo hosts
 --system-all, -m: Run emerge system on all Gentoo hosts
 --revdep-all, -v: Run revdep-rebuild on all Gentoo hosts
 --echall-all, -c: Check all Gentoo hosts for etc-updates
 --help, -h: Show this message

There a few things that need to be done before this code will work.
1) you need to have an user set up on each host, in my example I have called the use “maint” currently I use a password to access the hosts rather then ssh keys.
2) sudo to be installed anc configured like this :-

maint ALL = NOPASSWD: /usr/bin/emerge, /usr/bin/rsync, /usr/bin/revdep-rebuild, /usr/bin/find

3) A SQL database that can be accessed over the network, see the attached schema.

You will need to rename the files from .doc to to .rb and .sql as wordpress won’t allow me upload anything else.
Gentoo Host ruby script
Gentoo Host sql schema 

Advertisements

Burning XGD3 and XGD2 xbox360 backups on Linux

Posted in Code, HowTo on 25/02/2012 by Undersys

There are a lot of guides out there on how to burn xbox360 backups under windows.
Well I don’t use windows.. and after fucking a lot of dual layer DVD’s I though a post was in order.

Firstly to have a correct XGD3 backup you need a Lite-on burner with the iXtream BurnerMax firmware. Find that your self..

Secondly you should be running abgx360 on all your backups prior to burning them.  They have a Linux version, make sure you have “xterm” installed to as it will use that for console output.

I wrote this small script that uses growisofs to burn xbox 360 backups to my Lite-on external burner.
Usage is simple:-
# ./xburn.sh xgd3 filename.iso
replace xgd3 with xgd2 depending on what your burning.
Edit the DVD variable to your dvd-rw device.
Using this script and the below growisofs settings along side abgx360 i have had 100% success rate.

#!/bin/bash

#
# Version: 1.0.0
# Date: 20120225
# Author: undersys@undersys.net
# function: burn xbox 360 games with the correct layer break
# xgd2 layerbreak is 1913760, xgd3 is 2133520
# notes for growisfo :-
# -use-the-force-luke=notray = don't reload tray
# -use-the-force-luke=dao = burn disk at once
# -use-the-force-luke=break = manual set layer break
# Vars

# Chage this to match the dvd-rw device
DVD=/dev/sr0

DTYPE=$1
INAME=$2
# Functions

function testargs {
 if [[ -z $DTYPE ]]; then
 echo You must specify a disk type
 echo EG: xgd3 or xgd2
 exit
 fi

if [[ -z $INAME ]]; then
 echo You must specify a iso file name
 echo EG: /tmp/backup-game.iso
 exit
 fi
}

function burn {
 if [[ $DTYPE == xgd3 ]]; then
 echo Burning $INAME as $DTYPE with layerbreak 2133520..
 growisofs -use-the-force-luke=notray -use-the-force-luke=break:2133520 -speed=1 -Z $DVD=$INAME
 echo Done! Ejecting..
 eject $DVD
 else
 echo Burning $INAME as $DTYPE with layerbreak 1913760..
 growisofs -use-the-force-luke=notray -use-the-force-luke=dao -use-the-force-luke=break:1913760 -dvd-compat -speed=2 -Z $DVD=$INAME
 echo Done! Ejecting..
 eject $DVD
 fi
}
# Main

testargs
burn

# End

OpenVMS remove expired licences from LMF

Posted in Code, HowTo on 06/10/2011 by Undersys

If your renewing your OpenVMS PAK’s you will need to remove the expired licences.

I found a nice DCL script for this :-
LMF clean up

Hear is the DCL script :-

$! File: LMF_CLEANUP.COM
$!
$! Procedure to remove expired keys from the LMF database.
$!
$! Use this at your own risk!
$!
$! 25-Nov-2001	J. Malmberg	Original
$!
$!===========================================================================
$!
$pid = f$getjpi("","PID")
$temp_file = "sys$scratch:lmf_cleanup_''pid'.temp"
$on control_y then goto all_exit
$!
$! Backup the LMF database in case of oops
$!-----------------------------------------
$copy/log lmf$license: sys$scratch:lmf_backup.ldb
$write sys$output "lmf$license backed up to sys$scratch:lmf_backup.ldb"
$write sys$Output "You can delete this file after verifying the cleanup."
$!
$! Dump out the LMF records
$!-------------------------
$if f$search(temp_file) .nes. "" then delete 'temp_file';*
$license list/full/out='temp_file'
$!
$today = f$cvtime("","COMPARISON","DATE")
$pak_issuer = ""
$!
$open/read/end=loop_end tf 'temp_file'
$loop:
$   read tf line_in/error=loop_end
$   line_in = f$edit(line_in,"COMPRESS,TRIM,UPCASE")
$   key = f$edit(f$element(0, " ", f$element(0, ":", line_in)),"TRIM")
$   value = f$edit(f$element(1, ":", line_in),"TRIM")
$   if key .eqs. "ISSUER"
$   then
$	pak_issuer = value
$	pak_authorization = ""
$	pak_product = ""
$	pak_producer = ""
$	pak_term = ""
$   endif
$   if key .eqs. "AUTHORIZATION" then pak_authorization = value
$   if key .eqs. "PRODUCT" then pak_product = value
$   if key .eqs. "PRODUCER" then pak_producer = value
$   if key .eqs. "PAK"
$   then
$!	write sys$output "''pak_product', Date= ''value'"
$       if value .eqs. "(NONE)" then goto loop
$	pak_term = f$cvtime(value,"COMPARISON","DATE")
$!	show sym pak_term
$!	show sym today
$	if pak_term .lts. today
$	then
$	    write sys$output "Deleting License = ''pak_product', Date= ''value'"
$	    license delete 'pak_product'/auth='pak_authorization'-
			/producer='pak_producer'
$	endif
$   endif
$   goto loop
$loop_end:
$!
$! Clean up and exit
$!----------------------
$all_exit:
$if f$trnlnm("tf", "LNM$PROCESS_TABLE", "SUPERVISOR") .nes. ""
$then
$   close tf
$endif
$!if f$search(temp_file) .nes. "" then delete 'temp_file';*
$exit


Past this into a new file and execute it. If you get a message about unrecognized verb that's ok.

Zimbra Copy to Amazon S3

Posted in Code on 18/09/2011 by Undersys

I was kinda against using S3 for backups or anything as it has a cost.
By rotating my backups keeping under the gb store limit I manage to only pay for transfers and that’s always under 1$ USD.
Now I have 4 weeks of backups stored on S3 thanks to the below script that will :-
* Put the file to S3
* rotate the 4th oldest backup file and delete it
* email me the results.

First set up an S3 account,
second setup s3cmd tool (http://s3tools.org/s3cmd)

Edit the following to suit:-

LLOC=/var/log
PFILE=`/bin/cat /tmp/curzmbk`
BKBKT="s3://basstech/backups/"
DATE=`date +%Y-%m-%d`
BKLOC="/backup"
MAILADDRESS@SERVER

 

So here is the script to do it :-

#!/bin/bash

LLOC=/var/log
PFILE=`/bin/cat /tmp/curzmbk`
BKBKT="s3://bucket/backups/"
DATE=`date +%Y-%m-%d`
BKLOC="/backup"


function loghd() {
  echo "Copy and file rotation to S3" > $LLOC/S3-CPY-$DATE.log
  echo "Started at:- `date +%c`" >> $LLOC/S3-CPY-$DATE.log
  echo "--------------------------------------------" >> $LLOC/S3-CPY-$DATE.log
}

function lstbkt() {
  /usr/bin/s3cmd ls $@
}

function putflbkt() {
  echo  "Putting File $@" >> $LLOC/S3-CPY-$DATE.log
  /usr/bin/s3cmd put $BKLOC/$PFILE $@ >> $LLOC/S3-CPY-$DATE.log
  echo "Status  $?" >> $LLOC/S3-CPY-$DATE.log
}

function rotatebk() {
  echo "--------------------------------------------" >> $LLOC/S3-CPY-$DATE.log
  /usr/bin/s3cmd ls $BKBKT > /tmp/bkbktlst
  echo "Current Files in $BKBKT" >> $LLOC/S3-CPY-$DATE.log
  cat /tmp/bkbktlst >> $LLOC/S3-CPY-$DATE.log
  BKCNT=`/usr/bin/wc -l /tmp/bkbktlst | awk ' { print $1 }'`
  if [ $BKCNT -ge 3 ]; then
    echo "--------------------------------------------" >> $LLOC/S3-CPY-$DATE.log
    OLDBK=`cat /tmp/bkbktlst | sort | sed -n 1p | awk '{print $4}'`
    echo "Roating out oldest backup file:-" >> $LLOC/S3-CPY-$DATE.log
    echo "$OLDBK" >> $LLOC/S3-CPY-$DATE.log
    /usr/bin/s3cmd del $OLDBK  >> $LLOC/S3-CPY-$DATE.log
    echo "Status :- $?" >> $LLOC/S3-CPY-$DATE.log
    rm -f /tmp/bkbktlst
  else
    echo "Less then two backup files, not roating" >> $LLOC/S3-CPY-$DATE.log
    rm -f /tmp/bkbktlst
  fi
}

function lognd() {
  echo "--------------------------------------------" >> $LLOC/S3-CPY-$DATE.log
  echo "Copy to S3 ended on `date +%c`" >> $LLOC/S3-CPY-$DATE.log
  echo "--------------------------------------------" >> $LLOC/S3-CPY-$DATE.log
}

function mailrpt() {
  /bin/cat $LLOC/S3-CPY-$DATE.log | /bin/mail -s "S3 Copy Report for `date +%c`" MAILADDRESS@SERVER
}


loghd
putflbkt  $BKBKT
rotatebk
lognd
mailrpt

Zimbra Cold backup script.

Posted in Code on 18/09/2011 by Undersys

I don’t need a hot backup solution for Zimbra I do try and sleep at night…

So I wrote the following script that will do the following :-
* stop zimba
* tar the /opt zimbra
* check the tar ball against /opt/zimbra
* encrypt the tar ball
* Email out the results.

Edit the following to suit your install :-

BKDIR=/backup
ZMHOME=/opt/zimbra
TARTL=/
GPGUSR=zimbra
LLOC=/var/log
DESTINIATION@EMAIL
#!/bin/bash

BKDIR=/backup
ZMHOME=/opt/zimbra
TARTL=/
GPGUSR=zimbra
LLOC=/var/log

DATE=`date +%Y-%m-%d`
ZMBVER=`su - zimbra -c '/opt/zimbra/bin/zmcontrol -v | awk '\''{print $2}'\'''`

function loghd() {
  echo "Starting Zimbra Full Backup" > $LLOC/ZM-FB-$DATE.log
  echo "Backup started on `date +%c`" >> $LLOC/ZM-FB-$DATE.log
  echo "--------------------------------------------" >> $LLOC/ZM-FB-$DATE.log
}

function zmstart() {
  /bin/su - zimbra -c '/opt/zimbra/bin/zmcontrol start'
}

function zmstop() {
  /bin/su - zimbra -c '/opt/zimbra/bin/zmcontrol stop'
  echo "Stopping Zimbra" >> $LLOC/ZM-FB-$DATE.log
  echo "Status $?" >> $LLOC/ZM-FB-$DATE.log
  sleep 20
}

function zmstatus() {
  echo "--------------------------------------------" >> $LLOC/ZM-FB-$DATE.log
  echo "Zimbra Status :-" >> $LLOC/ZM-FB-$DATE.log
  /bin/su - zimbra -c '/opt/zimbra/bin/zmcontrol status' >> $LLOC/ZM-FB-$DATE.log
  sleep 20
}

function tarbk() {
  echo "--------------------------------------------" >> $LLOC/ZM-FB-$DATE.log
  echo "Starting backup on `date +%c`" >> $LLOC/ZM-FB-$DATE.log
  echo "Backing up $ZMHOME :-" >> $LLOC/ZM-FB-$DATE.log
  /bin/tar -zpcf $BKDIR/zimbra-$ZMBVER-$DATE.tgz $ZMHOME
  echo "Tar exit exist status $?" >> $LLOC/ZM-FB-$DATE.log
}

function chktar() {
  echo "--------------------------------------------" >> $LLOC/ZM-FB-$DATE.log
  echo "Checking status of tarball :-" >> $LLOC/ZM-FB-$DATE.log
  /bin/tar -df $BKDIR/zimbra-$ZMBVER-$DATE.tgz -C $TARTL >> $LLOC/ZM-FB-$DATE.log
  echo "Tarball integrity $?" >> $LLOC/ZM-FB-$DATE.log
}

function encbk() {
  echo "--------------------------------------------" >> $LLOC/ZM-FB-$DATE.log
  echo "Encripting file zimbra-$ZMBVER-$DATE.tgz" >> $LLOC/ZM-FB-$DATE.log
  echo "Output :- zimbra-$ZMBVER-$DATE.tgz.gpg" >> $LLOC/ZM-FB-$DATE.log
  /usr/bin/gpg -es -r $GPGUSR $BKDIR/zimbra-$ZMBVER-$DATE.tgz
  echo "GPG exit status $?" >> $LLOC/ZM-FB-$DATE.log
}

function killzm() {
  echo "--------------------------------------------" >> $LLOC/ZM-FB-$DATE.log
  echo "Killing all Zimbra owned process" >> $LLOC/ZM-FB-$DATE.log
  /usr/bin/pkill -9 -u zimbra
  echo "Kill status $?" >> $LLOC/ZM-FB-$DATE.log
}

function bkend() {
  echo "--------------------------------------------" >> $LLOC/ZM-FB-$DATE.log
  echo "Backup and GPG ended on `date +%c`" >> $LLOC/ZM-FB-$DATE.log
  echo "--------------------------------------------" >> $LLOC/ZM-FB-$DATE.log
  echo zimbra-$ZMBVER-$DATE.tgz.gpg > /tmp/curzmbk
}

function mailrpt() {
  /bin/cat $LLOC/ZM-FB-$DATE.log | /bin/mail -s "Backup Report for `date +%c`" DESTINIATION@EMAIL
}

loghd
zmstop
killzm
zmstatus
tarbk
chktar
encbk
zmstart
zmstatus
bkend
mailrpt

ps3vram and updated and optimized kernel for the PS3

Posted in Code, HowTo, Projects on 14/07/2009 by Undersys

So we left off with a working PS3 kernel.
An older version…
So  lets get that up to date. I will be using 2.6.30. You can find the .conf at the end of the post.
I am sure theres more that can be done to make this run faster on the PS3. Also I have not enabled the card readers yet.

To make use of the ps3vram as a high priority swap space I did the following.

  1. Create a file called /etc/ps3-vram-swap.sh
  2. Add the following to the file:-
    #!/bin/sh
    mkswap /dev/ps3vram  > /dev/null
    swapon -p 10 /dev/ps3vram  >/dev/null
  3. Make the file executable.
  4. Add the following line to  /etc/conf.d/local.start
    /etc/ps3-vram-swap.sh

Now the ps3vram will get added as priority 10 swap so it will get used before the HDD swap.
Output of swapon -s
Filename                Type        Size    Used    Priority
/dev/ps3da2                             partition    1686816    0    -1
/dev/ps3vram                            partition    250872    0    10

Hear is the .config file I did for the 2.6.30 kernel. I will keep working to make it better.
The PS3 is a LOT faster now compared to the kernel we build using the 2.6.23-ps3 source.
Keep in mind that its advisable to use main kernel.org sources now and not a patch-set.
Pretty much all PS3 patches from the early patch-sets have been merged.

Save the following as .config under /usr/src/linux . run make menuconfig if you need to edit it.
Rebuild your kernel with :-

  • make && make modules_install

Copy the vmlinux to your /boot and update kboot.conf.

Hear is a copy of my .config PS3 vmram Kernel .config
Save this file as a .txt its not a .doc file, wordpress is just been crap.

Script — Transencode Flac to Mp3 + tag rewrite

Posted in Code on 02/03/2009 by Undersys

The following script will trans encode FLAC to MP3 and read the VORBIS tags and write ID3 tags to the MP Will not alter original FLAC.

I don’t care about other tools!
Read the comments..
No i will not edit this for “your” needs
No i don’t care what happens to your system
Yes I will take constructive feedback!
You will need to scroll across the code block. I can not seem to make it wider right now.

Requires command line parameters in the following format :-
# ./flac2mp3.sh
No Source Dir given Please specify the full path to the source flac file

ALL CODE GPLv3

#!/bin/bash
#
# Desc :- Script will convert flac to mp3
#         Will pull flac tags into mp3 as ID3v2 tags
# Version :- 1.0.0
# Date :- 1-02-09
# Author :- undersys
# Dependencies :- lame-3.98.2, flac-1.2.1-r3, mp3gain-1.4.6-r2
#

# -------------------
# Set Global Config
# -------------------
#
# Only change seven settings below
#

# Set the base dir to write mp3s to, artist and album structure will be build under this
BASEDIR=/store/mp3

# Set who should own the files at the end
REALUSR=samba

# Set group to own files/folders
REALGRP=users

# Set path to lame binary
LAME=/usr/bin/lame

# Set path to flac binary
FLAC=/usr/bin/flac

# Set path to metaflac, part of flac
METAFLAC=/usr/bin/metaflac

# Set path to mp3gain binary
MP3GAIN=/usr/bin/mp3gain

# -------------------
# End Global Config
# -------------------

# -------------------
# Set Global Vars
# -------------------

# Must check for valid input first, befour we set any Global Vars
if [[ -z $1 ]]; then
        echo No Source Dir given
        echo Please specify the full path to the source flac files
        exit
fi

# Name of the file that holds file list
FLIST=flist.out

# Name of tags file
TAGSFILE=tags.out

# Find total coloums in input path
MAXCOL=`echo $1 | awk -F/ '{print NF}'`

# Select Album coloum
ALCOL=$(($MAXCOL - 1))

# Selet Artist coloum
ARCOL=$(($MAXCOL - 2))

# Select artist name from full path
ARTIST=`echo $1 | awk -F/ '{print $v1}' v1=$ARCOL`

# Select album name from full path
ALBUM=`echo $1 | awk -F/ '{print $v1}' v1=$ALCOL`

# Set source dir from input
SOURCEDIR=$1

# create a list of .flac files and write to file
FILES=`find $1 -name "*.flac" >$FLIST`

# ------------------
# End Global Vars
# ------------------

# --------------------
# Start Function Block
# --------------------

function cdir {
        if [[ -d $BASEDIR/$ARTIST/$ALBUM ]]; then
                echo This Album Has been encoded
                echo Script will now exit
                exit
        fi

        mkdir -p $BASEDIR/$ARTIST/$ALBUM
        }

function transcode {
        cp flist.out $BASEDIR/$ARTIST/$ALBUM/$FLIST
        cd $BASEDIR/$ARTIST/$ALBUM
        cat $BASEDIR/$ARTIST/$ALBUM/flist.out | while read line; do
        $METAFLAC --no-utf8-convert --export-tags-to=- ${line} | sed 's/=\(.*\)/="\1"/' >$TAGSFILE
        . ./$TAGSFILE
        FLACN=`echo ${line} | awk -F/ '{print$NF}'`
        MP3N=`echo $FLACN | sed 's/\..\{4\}$//'`
        $FLAC -d -c ${line} | $LAME -V3 --vbr-new \
        --tt "$TITLE" \
        --tn "$TRACKNUMBER" \
        --tg "$GENRE" \
        --ty "$DATE" \
        --tc "$COMMENT" \
        --ta "$ARTIST" \
        --tl "$ALBUM" \
        --add-id3v2 \
        - $MP3N.mp3
        done
       }

function replaygain {
        $MP3GAIN  -k -p -s r  $BASEDIR/$ARTIST/$ALBUM/*
        }

function setperms {
        chown -R "$REALUSR:$REALGRP" $BASEDIR/$ARTIST/$ALBUM
        chmod -R 774 $BASEDIR/$ARTIST/$ALBUM
        }

function clearvars {
        cat $BASEDIR/$ARTIST/$ALBUM/$TAGSFILE | while read line;
        do
        TAG=`echo "${line}" | awk -F= '{print $1}'` | unset $TAG
        done
        rm $BASEDIR/$ARTIST/$ALBUM/$TAGSFILE
        rm $BASEDIR/$ARTIST/$ALBUM/$FLIST
       }

# --------------------
# End Function Block
# --------------------

# --------------------
# main section
# --------------------

cdir
transcode
replaygain
setperms
clearvars

exit

# --------------------
# End main section
# --------------------

# -------------------
# End Script
# -------------------