Debian modifying permissions for files or directories

July 25th, 2008 No comments

Sometimes you want to change specific settings to either files or directories on your debian machine. To be able to do this I use the find command combined with the xargs command. Have a look at some possible commands:

find /share/ -type f -print0 | xargs -0 chmod 664
find /share/ -type d -print0 | xargs -0 chmod 775

The first line tries to find files only with -type f. And prints them to the stream with -print0 so xargs can process them with the -0 command. -print0 will put a NUL value between pathnames. This way paths that contain spaces can be parsed correctly by xargs.
I think it is pretty nifty and once again shows the power of the console!

Putting /var in ram

March 13th, 2008 1 comment

Well there you have it. I wanted to save some more power on my debian server. So I installed a flash disk and copied my debian install from hd to it.

After that I made my hd spin down automatically by using hdparm.

At the moment flash-memory is limited to ~10.000 write/erase cycles. By spreading the writes to disk the manufacturers try to avoid this problem. With a 24/7 server writing regularly to log files this might become a problem on the long run. That’s why I decided to put /var completely into memory by using a ramdisk. Note that you should not do/use this when you are running enterprise critical applications where a system crash might result in serious data loss.

First things first. First create a directory where we will persist our /var directory to in case of shutdown/reboot. I created a directory var-bak for this.

mkdir /var-bak

I then copied the /var directory to this /var-bak directory with cp -a.
After that I removed the /var-bak/run directory which should not be stored.

cp -a /var/* /var-bak/
rm -rf /var-bak/run

Ok, so now we have a copy of our contents in /var. Let’s mount the ramdisk. And copy the contents of /var-bak back to the ramdisk.

mount -t ramfs ramfs /var
cp -a /var-bak/* /var/

Ok there we have it. It’s all set up now. Using this method might cause some trouble. Try a df command to see what I mean.

router:~# df -h /var
Filesystem            Size  Used Avail Use% Mounted on
ramfs                    0     0     0   -  /var

That’s right it lists 0 as available space. Programs which perform space checks might report an ‘insufficient diskpace’ error. I had to fix the mysql init script to ignore this.

To be able to copy and synch the disks automatically on startup/shutdown/reboot I created a init script.
I called it ramdisk.sh and placed it in the /etc/init.d directory.

RSync makes sure the /var-bak directory keeps correctly synched with the /var directory. Also it makes sure the run directory is ignored during the synch process.

#! /bin/sh
# /etc/init.d/ramdisk.sh
#
 
case "$1" in
  start)
    echo "Copying files to Ramdisk"
    cp -a /var-bak/* /var/
    echo [`date +"%Y-%m-%d %H:%M"`] Ramdisk Synched from HD >> /var/log/ramdisk_sync.log
    ;;
  sync)
    echo "Synching files to Harddisk"
    echo [`date +"%Y-%m-%d %H:%M"`] Ramdisk Synched to HD >> /var/log/ramdisk_sync.log
    rsync -av --delete --exclude=run/ --exclude=run/** /var/ /var-bak/
    ;;
  stop)
    echo "Synching logfiles to Harddisk"
    echo [`date +"%Y-%m-%d %H:%M"`] Ramdisk Synched to HD >> /var/log/ramdisk_sync.log
    rsync -av --delete --exclude=run/ --exclude=run/** /var/ /var-bak/
    ;;
  *)
    echo "Usage: /etc/init.d/ramdisk.sh {start|stop|sync}"
    exit 1
    ;;
esac
 
exit 0

After that I wanted it to be started as early as possible. So I placed it as early in the process as possible.
Maybe this still needs some more tweaking but this works ok for me at this point. You can use the sync command to manually sync the ramdisk to disk.

update-rc.d ramdisk.sh defaults 00 99

That’s it. You could optionally run a cron job to synch the ramdisk every once in a while if you like. It will at least save quite some write cycles.

Maven and Java dependencies

March 6th, 2008 No comments

When building some Java projects with maven you might run into some missing Sun jars/artifacts.

[INFO] Failed to resolve artifact.
 
Missing:----------
 
1) javax.transaction:jta:jar:1.0.1B

Luckily you can fix this. :)

Go to the Java site and grab the jta-1_0_1B-classes.zip file.
Then manually import the file into your local maven repository by running the following command:

mvn install:install-file -Dfile=jta-1_0_1B-classes.zip -DgroupId=javax.transaction -DartifactId=jta -Dversion=1.0.1B  -Dpackaging=jar

Now you have it. If you wish you can add this dependency to your pom.xml manually now also.

<dependency>
    <groupid>javax.transaction</groupid>
    <artifactid>jta</artifactid>
    <version>1.0.1B</version>
</dependency>

For more information look at maven’s mini-howto.

Categories: Java Tags:

Linux: Mounting iso files

February 25th, 2008 No comments

Mounting iso images under linux is fairly straightforward. You can use the following command:

mount </path/to/isofile.iso> </mount/directory/location/> -t iso9660 -o loop

Categories: Linux Tags:

Subversion (SVN) Server Startup Script

February 12th, 2008 1 comment

To be able to start and stop my subversion server nicely I created the following script to startup and shutdown under debian.

#! /bin/sh
# /etc/init.d/svnserve: start and stop svnserve
 
# Exit immediately if a command exits with a nonzero exit status.
set -e
 
# svnserve exists and is executable
test -x /usr/bin/svnserve || exit 0
 
# Directory Where the Repository is located, created with svnadmin create
REPOS_DIR="/opt/svn"
 
# The pid-file
PIDFILE="/var/run/svnserve.pid"
 
case "$1" in
  start)
        echo -n "Starting Subversion (SVN) Server"
        start-stop-daemon --start --quiet \
                --exec /usr/bin/svnserve -- -dr $REPOS_DIR
        # Also tried the following line but it didn't work ok
        #       --make-pidfile --pidfile $PIDFILE
        PID=`pidof svnserve` || true
        echo $PID &gt; $PIDFILE
        echo "."
        ;;
  stop)
        echo -n "Stopping Subversion (SVN) Server"
        start-stop-daemon --stop --quiet --oknodo --pidfile $PIDFILE
        if [ -f $PIDFILE ]
        then
                rm $PIDFILE
        fi
        echo "."
        ;;
 
  *)
        echo "Usage: /etc/init.d/svnserve {start|stop}"
        exit 1
esac
 
exit 0

Place the script in /etc/init.d/svnserve and run the following command:

update-rc.d svnserve defaults

Enabling Wake-on-LAN on your local machine (Windows)

January 29th, 2008 1 comment

Well ain’t it all a bit nasty? You are at work and you left some important files on your home pc because you were working on it last night. You are starting to pull out your hair because you now either need to drive back to fetch it or do all your hard work again.

But not any more. We arrived in this new millennium and we want to automate everything as much as possible. (Well my inner geek at least ;) )

Let’s start with enabling Wake-on-LAN (WOL) on your home pc.

Well Wake-on-LAN already states it. You will need to boot from a LAN machine in order to make this work. I use my router to be able to do this trick. It is a Linksys WRT54GL with DD-WRT flashed on it. It will allow me to log in with ssh or putty and load the webinterface so I can wake up my pc. In this case an Asus P5K-E.

Before we can boot this baby we need to make sure WOL is enabled in the BIOS. So boot it and get into the bios by pressing whatever button you need to at startup.
Start to look for something like Wake-on-LAN or PME (Power Management Event) in your BIOS. Depending on your hardware it will have different settings. In my case I had to enable PME events on PCIE devices as my ethernet was a PCIE device.

After enabling it save your settings and boot into windows. (used for this example, for linux do a bit of research on the net)
With the most common options you only need to do a proper shutdown from windows in order to make your WOL work. In some cases you will have to do some extra work by configuring your ethernet device. I have some screenshots of this event from the control panel.

WOL1

WOL2

So go to your network connections. Right click on your connection and open the properties. Then configure your network adapter and search for Wake from Shutdown / Wake-on-LAN or something similar and enable it. Also make sure that your wake-up options is set to ‘Magic Packet’ and not ‘Pattern Matching’. It caused my pc to boot when I did not want it to boot. After that shutdown your machine and try to boot it from LAN. There are a lot of programs out in the wild being capable of sending magic packets across the network. For debian based machines there are etherwake /wakeonlan and for windows winwake / magic packet sender. A bit of searching on the net might get you some of the wake-up clients you need. Have fun!

Tomcat behind reverse proxy on Apache

January 23rd, 2008 1 comment

If you have apache installed as your main http server you might want all requests to be handled by apache. But what if you want to have a standalone tomcat install? Well this is still possible by using apache’s mod_proxy. You might need to install it if you are running debian. Do some ‘apt-cache search apache proxy’ abracadbra and install mod_proxy if it is not already installed. You might even need to enable it with ‘a2enmod proxy’.

Edit your configuration (I added it to the sites-available/default) and add something like this:

    # Tomcat Proxy
    RedirectMatch ^/tomcat$ /tomcat/
    ProxyRequests Off
    ProxyVia Off
    ProxyPass /tomcat/ http://localhost:8082/
    ProxyPassReverse /tomcat/ http://localhost:8082/

You also might want add some access rules to make sure noone will be able to abuse your proxy (especially when running a forward proxy).

<proxy *:80>
Order deny,allow
Deny from all
</proxy>
<proxy *:8080>
Order deny,allow
Deny from all
Allow from 127.0.0.1
</proxy>

This should do the trick (after reloading apache of course, /etc/init.d/apache2 reload). The redirectmatch will make sure that users that reach http://serveraddress/tomcat will be redirected to http://serveraddress/tomcat/ because otherwise they would not reach the page.
The proxypass points to /tomcat/ note the trailing slash. If you omit the trailing slash your images will not load correctly. port 8082 is tomcat’s proxy port which will allow proxied connect calls.

You can use the same trick for any other webserver you have running on another port. (webmin, azureus web html, etc)

Setting up an apache2 proxy server

January 15th, 2008 5 comments

note: I found out that you can also use the ssh -D option to have ssh function as a SOCKS server to get similar behaviour without the hassle of configuring apache.

Are you at work behind a big bad evil proxy? Afraid of your privacy? Set up a proxy on your local home server so you can browse safely.

What are we going to use:

  • Putty
  • Apache 2

Is that all we need? Yes that’s all. ;)

Ok let’s set up our apache 2 proxy first. It is a good idea to add some security to your proxy server so not everyone can reach it. You might want to restrict it to the localhost only. This tutorial is based on debian install of apache 2. So hang on and let’s go.

First change the ports apache2 is listening in to. Edit /etc/apache2/ports.conf for this purpose and add the line:

Listen 8080

This will make the apache2 server listen to port 8080. If there is a line that makes apache2 listen on port 443 (https) you might want to disable it. We are going to use putty to connect to this port.

You might need to download mod_proxy for apache2 to be able to use proxying. If it is not already enabled use:

a2enmod proxy
a2enmod proxy_connect
a2enmod proxy_html
a2enmod proxy_ftp

This will enable it. If mod proxy is not yet installed at all use an apt-cache search mod proxy to locate and install it through apt.
the mod proxy_connect is required to be able to handle SSL calls through your proxy.

Create a new entry in /etc/apache/sites-available and name it ‘proxy’ for example. Insert something like the following:

<virtualhost *:8080>
        ServerAdmin webmaster@localhost
 
        ProxyRequests On
        ProxyVia On
 
        #Add ports you want to be able to connect to through your proxy here
        AllowCONNECT 443 563 1863
        #443   = SSL
        #563   = TLS
        #1863  = MSN Messenger
 
        DocumentRoot /var/www/
        <Directory />
                Options FollowSymLinks
                AllowOverride None
        </Directory>
 
        ErrorLog /var/log/apache2/proxy-error.log
        TransferLog /var/log/apache2/proxy-transfer.log
        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel notice
</virtualhost>

Enable this new site by typing:

a2ensite proxy

It would be really wise to limit the access to your forward proxy.
edit the proxy.conf file in /mods-available/proxy.conf.
Add something like this to allow only connections from localhost:

<proxy *:80>
    Order Deny,Allow
    Deny from all
</proxy>
<proxy *:8080>
    Order Deny,Allow
    Deny from all
    Allow from 127.0.0.1
</proxy>

reload the webserver after this by running:

/etc/init.d/apache2 reload

Add 443 to your ssh listen ports by opening /etc/ssh/sshd_config.
Edit it so it reads:

# What ports, IPs and protocols we listen for
Port 22
Port 443

Restart the ssh daemon by calling:

/etc/init.d/ssh restart

Now you are ready to go. You can try your proxy now by using putty to connect to your server and tunnel port 8080 to another port on your local machine. It might also be a good idea to enable zip compression on your connection (Putty:Connection->SSH->Enable Compression) to speed things up a bit.
Now you can use firefox or another app and connect on localhost:[bound putty port] to connect to your proxy.

If you want firefox to do the dns lookups on the remote end you should open your about:config page by typing this in the address bar. Lookup the value:

network.proxy.socks_remote_dns

Set the value to true to do remote dns lookups.

Setting up Rsync

January 14th, 2008 No comments

I was getting fed up with having to synchronize several folders containing photos. So I was looking for a good way to synchronize these images across the network. I first tried to put the images into Subversion but this turned out to be overkill. So that’s why I put my focus on Rsync.

Rsync has support for incremental file exchange. Not that I am going to use it but it still is cool. I will only use Rsync to keep my photo directories synched with the server. So lets install it.

apt-GET install rsync

Edit the rsyncd.conf file in the /etc directory and create some entries here. Example is my photos entry here.

#/etc/rsyncd.conf
#file containing username+passwords IN the form <name>:
<password> the file should be readable ONLY BY USER OR GROUP.
#so chmod 660 OR 600 would be necessary.
secrets file = /etc/rsyncd.secrets
motd file = /etc/rsyncd.motd 
 
#Below are actually defaults, but TO be ON the safe side...
READ ONLY = yes
list = yes
# WITH what USER permissions should rsync handle directories?
uid = nobody
#gid = nobody
#On debian GROUP nobody IS mapped TO nogroup
gid = nogroup
#if stricts mode IS TRUE secrets file should NOT be readable BY ALL. TO disable this turn stricts mode TO off. (built IN FOR cygwin users)
strict modes = TRUE
 
[photos]
comment = Family Photos
path = /share/photos
auth users = wytze,wytske
READ ONLY = no
hosts allow = 192.168.*
hosts deny = *
list = FALSE
</password></name>

Start the rsync daemon:

rsync --daemon

Now you can start synchronizing. Synchronizing works like copying files with scp. So it’s fairly straightforward.

To retrieve a list of entries if listing is enabled:

rsync -avz wytze@debian::

To synchronize files from server:

rsync -avz wytze@debian::photos /opt/my-local-photos

To synchronize files to the server:

rsync -avz /opt/my-local-photos/ wytze@debian::photos

Please mind the trailing slashes. Try out the difference with and without the trailing slash. Pretty straightforward there.

Categories: General Tags:

pl/sql splitting strings/varchar by delimiter

January 11th, 2008 No comments

I created the following function for oracle with pl/sql so Strings/varchar items can be split and put into a varray.

CREATE OR REPLACE TYPE my_array IS varray(1000) OF VARCHAR2(255);
 
CREATE OR REPLACE FUNCTION my_split(p_string IN VARCHAR2, p_delim IN VARCHAR2)
RETURN my_array
AS
    p_last_index NUMBER := 1;
    p_current_index NUMBER := 1;
 
    p_array_pointer NUMBER := 1;
    p_items my_array := my_array();
    p_item VARCHAR2(255);
BEGIN
     -- get index of split character
     p_last_index := INSTR(p_string,p_delim,p_current_index,1);
 
     -- while split characters are found
     -- add it to the varray
     WHILE( p_last_index > 0 ) LOOP
    	-- get first item
        p_item := SUBSTR(p_string, p_current_index, (p_last_index - p_current_index));
 
	-- put item in varray
        p_items.extend;
        p_items(p_array_pointer) := p_item;
        p_array_pointer := p_array_pointer + 1;
 
    	-- update indexes
        p_current_index := p_last_index + LENGTH(p_delim);
        p_last_index := INSTR(p_string,p_delim,p_current_index,1);
    END LOOP;
 
    -- get last item
    p_item := SUBSTR(p_string, p_current_index);
    -- put item in varray
    p_items.extend;
    p_items(p_array_pointer) := p_item;
 
    --dbms_output.put_line(substr('Value of p_receiver='||p_receiver,1,255));
 
    /*
    Example of how to loop through the items:
 
    for a_index in 1..p_items.count loop
    	dbms_output.put_line(substr('Value of array('||a_index||'):'||p_items(a_index),1,255));
    end loop;
    */
 
    RETURN p_items;
END;

Small example how you can use this now:

DECLARE
	test my_array;
	input VARCHAR2(255) := 'a;b;c';
	delim VARCHAR2(1) := ';';
BEGIN
	test := my_split(input,delim);
 
	FOR a_index IN 1..test.COUNT LOOP
		DBMS_OUTPUT.put_line(SUBSTR('Value of array('||a_index||'):'||test(a_index),1,255));
	END LOOP;
END;
Categories: Coding Tags: